[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:04 dist-test-slave-9jll krb5kdc[2436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:04 dist-test-slave-9jll krb5kdc[2436](info): set up 2 sockets
Sep 27 18:38:04 dist-test-slave-9jll krb5kdc[2436](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 18:38:05.801242 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:38:05 dist-test-slave-9jll krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:38:05.996640 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:35161
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:35161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:06.385124 2449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:06.385494 2449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:06.385629 2449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:06.393872 2449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:06.394016 2449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:06.394063 2449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:06.394124 2449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:06.394170 2449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:06.406370 2449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:35161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:35161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:06.408318 2449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:06.410040 2449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:06.424467 2454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:06.424422 2455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:06.424386 2457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:06.432070 2449 server_base.cc:1034] running on GCE node
I20260927 18:38:06.433312 2449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:06.435421 2449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:06.436673 2449 hybrid_clock.cc:648] HybridClock initialized: now 1790534286436624 us; error 48 us; skew 500 ppm
I20260927 18:38:06.437122 2449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:06 dist-test-slave-9jll krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534286, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:06.440882 2449 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:38:06.442910 2449 webserver.cc:460] Webserver started at http://127.2.95.126:40083/ using document root <none> and password file <none>
I20260927 18:38:06.443810 2449 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:06.443938 2449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:06.444295 2449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:06.447996 2449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "d46fe6f600ba417faceeb635386a8c02"
format_stamp: "Formatted at 2026-09-27 18:38:06 on dist-test-slave-9jll"
server_key: "bb4136e23b51faf250bb5c1b363a8f01"
server_key_iv: "e42d0d150b94f6d1fd3acf096ccc2d3f"
server_key_version: "encryptionkey@0"
I20260927 18:38:06.448874 2449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "d46fe6f600ba417faceeb635386a8c02"
format_stamp: "Formatted at 2026-09-27 18:38:06 on dist-test-slave-9jll"
server_key: "bb4136e23b51faf250bb5c1b363a8f01"
server_key_iv: "e42d0d150b94f6d1fd3acf096ccc2d3f"
server_key_version: "encryptionkey@0"
I20260927 18:38:06.455416 2449 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 18:38:06.459793 2464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:06.461603 2449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:38:06.461849 2449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "d46fe6f600ba417faceeb635386a8c02"
format_stamp: "Formatted at 2026-09-27 18:38:06 on dist-test-slave-9jll"
server_key: "bb4136e23b51faf250bb5c1b363a8f01"
server_key_iv: "e42d0d150b94f6d1fd3acf096ccc2d3f"
server_key_version: "encryptionkey@0"
I20260927 18:38:06.462074 2449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:06.487188 2449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:06.492817 2449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:06.493175 2449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:06.514833 2449 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:35161
I20260927 18:38:06.514856 2515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:35161 every 8 connection(s)
I20260927 18:38:06.516539 2449 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:06.519846 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 2449
I20260927 18:38:06.520205 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:06.521353 2516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:06.522471 2429 external_mini_cluster.cc:1428] Setting key 916b1cc8117bd0d87a9176311c10a52b
I20260927 18:38:06.535095 2516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02: Bootstrap starting.
I20260927 18:38:06.539379 2516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:06.540802 2516 log.cc:824] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:06.544617 2516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02: No bootstrap required, opened a new log
I20260927 18:38:06.552896 2516 raft_consensus.cc:348] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46fe6f600ba417faceeb635386a8c02" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 35161 } }
I20260927 18:38:06.553373 2516 raft_consensus.cc:374] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:06.553472 2516 raft_consensus.cc:729] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d46fe6f600ba417faceeb635386a8c02, State: Initialized, Role: FOLLOWER
I20260927 18:38:06.554219 2516 consensus_queue.cc:260] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46fe6f600ba417faceeb635386a8c02" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 35161 } }
I20260927 18:38:06.554471 2516 raft_consensus.cc:388] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:06.554613 2516 raft_consensus.cc:482] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:06.554783 2516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 18:38:06 dist-test-slave-9jll krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:06.557550 2516 raft_consensus.cc:504] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46fe6f600ba417faceeb635386a8c02" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 35161 } }
I20260927 18:38:06.558200 2516 leader_election.cc:302] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d46fe6f600ba417faceeb635386a8c02; no voters:
I20260927 18:38:06.558964 2516 leader_election.cc:288] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:06.559233 2521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:06.560040 2521 raft_consensus.cc:686] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [term 1 LEADER]: Becoming Leader. State: Replica: d46fe6f600ba417faceeb635386a8c02, State: Running, Role: LEADER
I20260927 18:38:06.560770 2521 consensus_queue.cc:237] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46fe6f600ba417faceeb635386a8c02" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 35161 } }
I20260927 18:38:06.561753 2516 sys_catalog.cc:564] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:06.570500 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:06.527887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.2.95.126:35161)
0927 18:38:06.528885 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:06.528904 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:06.530227 (+ 1323us) server_negotiation.cc:407] Connection header received
0927 18:38:06.541501 (+ 11274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:06.541555 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:06.542255 (+ 700us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:06.542825 (+ 570us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:06.546400 (+ 3575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:06.547860 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:06.549700 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:06.550460 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:06.550680 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:06.557570 (+ 6890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:06.557583 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:06.557602 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:06.557652 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:06.562260 (+ 4608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:06.563712 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:06.563724 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:06.563737 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:06.564037 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:06.564745 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:06.564751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:06.564755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:06.565705 (+ 950us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:06.566078 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:06.567327 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0927 18:38:06.567914 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":442,"threads_started":1}
I20260927 18:38:06.571463 2522 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d46fe6f600ba417faceeb635386a8c02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46fe6f600ba417faceeb635386a8c02" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 35161 } } }
I20260927 18:38:06.572649 2522 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:06.572304 2523 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46fe6f600ba417faceeb635386a8c02. Latest consensus state: current_term: 1 leader_uuid: "d46fe6f600ba417faceeb635386a8c02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46fe6f600ba417faceeb635386a8c02" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 35161 } } }
I20260927 18:38:06.573033 2523 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:06.579638 2531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:06.585525 2531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:06.596077 2531 catalog_manager.cc:1347] Generated new cluster ID: f9ce87c50d0240f5a06efe6e9680cf4c
I20260927 18:38:06.596231 2531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:06.609517 2531 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:06.611337 2531 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:06.620076 2531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d46fe6f600ba417faceeb635386a8c02: Generated new TSK 0
I20260927 18:38:06.621656 2531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:38:06.678141 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:35161
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:07.075147 2542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:07.075531 2542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:07.075642 2542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:07.084613 2542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:07.084831 2542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:07.084985 2542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:38:07.098253 2542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:35161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:07.100086 2542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:07.101895 2542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:07.117803 2547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:07.117911 2548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:07.118242 2550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:07.119251 2542 server_base.cc:1034] running on GCE node
I20260927 18:38:07.120242 2542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:07.121582 2542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:07.122922 2542 hybrid_clock.cc:648] HybridClock initialized: now 1790534287122856 us; error 57 us; skew 500 ppm
I20260927 18:38:07.123368 2542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:07 dist-test-slave-9jll krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534287, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:07.127560 2542 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:38:07.129814 2542 webserver.cc:460] Webserver started at http://127.2.95.65:46797/ using document root <none> and password file <none>
I20260927 18:38:07.130793 2542 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:07.130946 2542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:07.131331 2542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:07.135789 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "f4bc72eb28364089901d8dd5b57ea25a"
format_stamp: "Formatted at 2026-09-27 18:38:07 on dist-test-slave-9jll"
server_key: "b8e429528af9fc8cf070de081abd9b8a"
server_key_iv: "d56a835361a56f6ac3e7aa91fd6feb56"
server_key_version: "encryptionkey@0"
I20260927 18:38:07.137121 2542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "f4bc72eb28364089901d8dd5b57ea25a"
format_stamp: "Formatted at 2026-09-27 18:38:07 on dist-test-slave-9jll"
server_key: "b8e429528af9fc8cf070de081abd9b8a"
server_key_iv: "d56a835361a56f6ac3e7aa91fd6feb56"
server_key_version: "encryptionkey@0"
I20260927 18:38:07.145112 2542 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.007s
I20260927 18:38:07.150099 2557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:07.152355 2542 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:38:07.152632 2542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "f4bc72eb28364089901d8dd5b57ea25a"
format_stamp: "Formatted at 2026-09-27 18:38:07 on dist-test-slave-9jll"
server_key: "b8e429528af9fc8cf070de081abd9b8a"
server_key_iv: "d56a835361a56f6ac3e7aa91fd6feb56"
server_key_version: "encryptionkey@0"
I20260927 18:38:07.152942 2542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:07.170826 2542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:07.178069 2542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:07.178630 2542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:07.180117 2542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:07.181833 2542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:07.181921 2542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:07.182053 2542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:07.182111 2542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:07.218235 2542 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:33189
I20260927 18:38:07.218291 2669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:33189 every 8 connection(s)
I20260927 18:38:07.220021 2542 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:07.221218 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 2542
I20260927 18:38:07.221498 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:07.222286 2429 external_mini_cluster.cc:1428] Setting key 92ce0378a0d3d6a6da5af4223097b1a0
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Sep 27 18:38:07 dist-test-slave-9jll krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534287, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:07.256131 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:07.223319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:46789 (local address 127.2.95.126:35161)
0927 18:38:07.223897 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:07.223903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:07.225647 (+ 1744us) server_negotiation.cc:407] Connection header received
0927 18:38:07.228820 (+ 3173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:07.228826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:07.228919 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:07.229037 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:07.232428 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:07.233362 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:07.235324 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:07.235805 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:07.236043 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:07.248530 (+ 12487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:07.248538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:07.248550 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:07.248607 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:07.251876 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:07.253118 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:07.253123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:07.253126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:07.253281 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:07.253891 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:07.253896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:07.253898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:07.254304 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:07.254439 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:07.255523 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0927 18:38:07.255755 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":314,"threads_started":1}
I20260927 18:38:07.257942 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:07.224026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:35161 (local address 127.2.95.65:46789)
0927 18:38:07.225151 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:07.225222 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:07.228329 (+ 3107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:07.229493 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:07.229508 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:07.230962 (+ 1454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:07.232151 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:07.232185 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:07.233574 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:07.233578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:07.235132 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:07.235149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:07.236156 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:07.236167 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:07.236488 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:07.245268 (+ 8780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:07.245305 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:07.248101 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:07.252116 (+ 4015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:07.252129 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:07.252166 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:07.252896 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:07.253399 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:07.253404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:07.253407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:07.253724 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:07.254443 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:07.254453 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:07.255003 (+ 550us) client_negotiation.cc:769] Sending connection context
0927 18:38:07.255360 (+ 357us) client_negotiation.cc:240] Negotiation successful
0927 18:38:07.256034 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":420,"threads_started":1}
I20260927 18:38:07.260272 2670 heartbeater.cc:347] Connected to a master server at 127.2.95.126:35161
I20260927 18:38:07.260700 2670 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:07.261656 2670 heartbeater.cc:511] Master 127.2.95.126:35161 requested a full tablet report, sending...
I20260927 18:38:07.264648 2481 ts_manager.cc:194] Registered new tserver with Master: f4bc72eb28364089901d8dd5b57ea25a (127.2.95.65:33189)
I20260927 18:38:07.266767 2481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:46789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:38:07.273559 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:35161
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:07.685075 2676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:07.685447 2676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:07.685582 2676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:07.694677 2676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:07.694839 2676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:07.695008 2676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:38:07.707069 2676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:35161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:07.708969 2676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:07.710834 2676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:07.726863 2681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:07.726820 2682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:07.726902 2684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:07.727653 2676 server_base.cc:1034] running on GCE node
I20260927 18:38:07.728399 2676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:07.729565 2676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:07.730794 2676 hybrid_clock.cc:648] HybridClock initialized: now 1790534287730736 us; error 54 us; skew 500 ppm
I20260927 18:38:07.731165 2676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:07 dist-test-slave-9jll krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534287, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:07.734777 2676 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:38:07.736550 2676 webserver.cc:460] Webserver started at http://127.2.95.66:41227/ using document root <none> and password file <none>
I20260927 18:38:07.737386 2676 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:07.737495 2676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:07.738384 2676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:07.742442 2676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "39d64db46852438b9980fe544cda2636"
format_stamp: "Formatted at 2026-09-27 18:38:07 on dist-test-slave-9jll"
server_key: "ea04e4c1d42bd6ba3fe95afa5b19632d"
server_key_iv: "ee12306c7bdaa13028dd7a04a5c9fa08"
server_key_version: "encryptionkey@0"
I20260927 18:38:07.743345 2676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "39d64db46852438b9980fe544cda2636"
format_stamp: "Formatted at 2026-09-27 18:38:07 on dist-test-slave-9jll"
server_key: "ea04e4c1d42bd6ba3fe95afa5b19632d"
server_key_iv: "ee12306c7bdaa13028dd7a04a5c9fa08"
server_key_version: "encryptionkey@0"
I20260927 18:38:07.750772 2676 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 18:38:07.755360 2691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:07.757380 2676 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:38:07.757570 2676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "39d64db46852438b9980fe544cda2636"
format_stamp: "Formatted at 2026-09-27 18:38:07 on dist-test-slave-9jll"
server_key: "ea04e4c1d42bd6ba3fe95afa5b19632d"
server_key_iv: "ee12306c7bdaa13028dd7a04a5c9fa08"
server_key_version: "encryptionkey@0"
I20260927 18:38:07.757866 2676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:07.786550 2676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:07.793100 2676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:07.793421 2676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:07.794689 2676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:07.796329 2676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:07.796413 2676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:07.796510 2676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:07.796558 2676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:07.834689 2676 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:39913
I20260927 18:38:07.834787 2803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:39913 every 8 connection(s)
I20260927 18:38:07.836233 2676 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:07.837982 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 2676
I20260927 18:38:07.838182 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:07.838703 2429 external_mini_cluster.cc:1428] Setting key c02eceebfe01fc9015c370d071334907
Sep 27 18:38:07 dist-test-slave-9jll krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534287, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
I20260927 18:38:07.860637 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:07.839507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:39421 (local address 127.2.95.126:35161)
0927 18:38:07.840054 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:07.840062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:07.841402 (+ 1340us) server_negotiation.cc:407] Connection header received
0927 18:38:07.843580 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:07.843586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:07.843683 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:07.843787 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:07.845867 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:07.846679 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:07.848305 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:07.848912 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:07.849043 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:07.853184 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:07.853193 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:07.853199 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:07.853246 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:07.856823 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:07.857731 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:07.857737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:07.857741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:07.857896 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:07.858542 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:07.858547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:07.858550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:07.858893 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:07.859018 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:07.860020 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0927 18:38:07.860318 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":259,"threads_started":1}
I20260927 18:38:07.862006 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:07.840294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:35161 (local address 127.2.95.66:39421)
0927 18:38:07.841226 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:07.841280 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:07.843269 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:07.844073 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:07.844084 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:07.844856 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:07.845624 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:07.845644 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:07.846835 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:07.846840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:07.848160 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:07.848172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:07.849170 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:07.849176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:07.849408 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:07.850247 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:07.850272 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:07.852904 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:07.857006 (+ 4102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:07.857015 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:07.857034 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:07.857553 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:07.858087 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:07.858093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:07.858097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:07.858373 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:07.859038 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:07.859046 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:07.859555 (+ 509us) client_negotiation.cc:769] Sending connection context
0927 18:38:07.859859 (+ 304us) client_negotiation.cc:240] Negotiation successful
0927 18:38:07.860394 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":373,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
I20260927 18:38:07.864805 2804 heartbeater.cc:347] Connected to a master server at 127.2.95.126:35161
I20260927 18:38:07.865253 2804 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:07.866248 2804 heartbeater.cc:511] Master 127.2.95.126:35161 requested a full tablet report, sending...
I20260927 18:38:07.868314 2481 ts_manager.cc:194] Registered new tserver with Master: 39d64db46852438b9980fe544cda2636 (127.2.95.66:39913)
I20260927 18:38:07.869378 2481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:39421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:38:07.890764 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:35161
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:38:08.271057 2670 heartbeater.cc:503] Master 127.2.95.126:35161 was elected leader, sending a full tablet report...
W20260927 18:38:08.298310 2810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:08.298695 2810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:08.298826 2810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:08.307593 2810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:08.307763 2810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:08.307931 2810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:38:08.319823 2810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:35161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:08.322067 2810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:08.323803 2810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:08.340830 2818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:08.340790 2816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:08.340752 2815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:08.342600 2810 server_base.cc:1034] running on GCE node
I20260927 18:38:08.343422 2810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:08.344632 2810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:08.346047 2810 hybrid_clock.cc:648] HybridClock initialized: now 1790534288345944 us; error 55 us; skew 500 ppm
I20260927 18:38:08.346621 2810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:08 dist-test-slave-9jll krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534288, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:08.350612 2810 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:38:08.352453 2810 webserver.cc:460] Webserver started at http://127.2.95.67:33257/ using document root <none> and password file <none>
I20260927 18:38:08.353317 2810 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:08.353447 2810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:08.353868 2810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:08.357983 2810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "ce79701198dd455faae88a9ff81ad703"
format_stamp: "Formatted at 2026-09-27 18:38:08 on dist-test-slave-9jll"
server_key: "cf84c1c8e6a272db723ee94c75bb4fb6"
server_key_iv: "4f32329c2b8ee016035a1a2c2af77dcd"
server_key_version: "encryptionkey@0"
I20260927 18:38:08.359054 2810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "ce79701198dd455faae88a9ff81ad703"
format_stamp: "Formatted at 2026-09-27 18:38:08 on dist-test-slave-9jll"
server_key: "cf84c1c8e6a272db723ee94c75bb4fb6"
server_key_iv: "4f32329c2b8ee016035a1a2c2af77dcd"
server_key_version: "encryptionkey@0"
I20260927 18:38:08.366614 2810 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260927 18:38:08.371565 2825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:08.373430 2810 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:38:08.373772 2810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "ce79701198dd455faae88a9ff81ad703"
format_stamp: "Formatted at 2026-09-27 18:38:08 on dist-test-slave-9jll"
server_key: "cf84c1c8e6a272db723ee94c75bb4fb6"
server_key_iv: "4f32329c2b8ee016035a1a2c2af77dcd"
server_key_version: "encryptionkey@0"
I20260927 18:38:08.374038 2810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:08.399466 2810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:08.404901 2810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:08.405257 2810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:08.406530 2810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:08.408262 2810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:08.408349 2810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:08.408488 2810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:08.408542 2810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:08.444761 2810 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:35347
I20260927 18:38:08.444813 2937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:35347 every 8 connection(s)
I20260927 18:38:08.446309 2810 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:38:08.454396 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 2810
I20260927 18:38:08.454639 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:08.455246 2429 external_mini_cluster.cc:1428] Setting key e5aeebe2cc8858f15814c3665f91659c
Sep 27 18:38:08 dist-test-slave-9jll krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534288, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:08.470752 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:08.449383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:34147 (local address 127.2.95.126:35161)
0927 18:38:08.449995 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:08.450002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:08.451149 (+ 1147us) server_negotiation.cc:407] Connection header received
0927 18:38:08.453185 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:08.453189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:08.453286 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:08.453392 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:08.455939 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.457014 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.459285 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.459797 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.459906 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:08.463937 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:08.463944 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:08.463947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:08.463989 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:08.466945 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:08.467944 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:08.467950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:08.467954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:08.468117 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:08.468655 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:08.468659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:08.468661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:08.469049 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:08.469188 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:08.470188 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0927 18:38:08.470385 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":310,"threads_started":1}
I20260927 18:38:08.472316 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:08.450094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:35161 (local address 127.2.95.67:34147)
0927 18:38:08.450940 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:08.450995 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:08.452877 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:08.453725 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:08.453738 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:08.454697 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:08.455676 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:08.455707 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.457206 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.457210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:08.459116 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:08.459131 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.460018 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.460026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:08.460303 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:08.460999 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:08.461024 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:08.463655 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:08.467156 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:08.467165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:08.467184 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:08.467743 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:08.468243 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:08.468247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:08.468250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:08.468480 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:08.469206 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:08.469214 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:08.469659 (+ 445us) client_negotiation.cc:769] Sending connection context
0927 18:38:08.470009 (+ 350us) client_negotiation.cc:240] Negotiation successful
0927 18:38:08.470688 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":335,"threads_started":1}
I20260927 18:38:08.474128 2938 heartbeater.cc:347] Connected to a master server at 127.2.95.126:35161
I20260927 18:38:08.474449 2938 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:08.475476 2938 heartbeater.cc:511] Master 127.2.95.126:35161 requested a full tablet report, sending...
I20260927 18:38:08.477177 2481 ts_manager.cc:194] Registered new tserver with Master: ce79701198dd455faae88a9ff81ad703 (127.2.95.67:35347)
I20260927 18:38:08.478147 2481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:34147
I20260927 18:38:08.486368 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:38:08 dist-test-slave-9jll krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534288, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:38:08 dist-test-slave-9jll krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534288, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:08.536471 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:08.518877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.2.95.65:33189)
0927 18:38:08.519912 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:08.519921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:08.519954 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:38:08.520040 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:08.520049 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:08.520323 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:08.520469 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:08.522480 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.523693 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.525054 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.525764 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.525865 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:08.530347 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:08.530363 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:08.530383 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:08.530432 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:08.533548 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:08.534409 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:08.534420 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:08.534423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:08.534596 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:08.534964 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:08.534968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:08.534969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:08.535352 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:08.535625 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:08.535908 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 18:38:08.536144 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":702,"threads_started":1}
W20260927 18:38:08.538074 2585 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35086
Sep 27 18:38:08 dist-test-slave-9jll krb5kdc[2436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:38:08 dist-test-slave-9jll krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:08.587011 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:08.571510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.2.95.65:33189)
0927 18:38:08.571824 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:08.571832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:08.572066 (+ 234us) server_negotiation.cc:407] Connection header received
0927 18:38:08.572254 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:08.572258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:08.572340 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:08.572428 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:08.574031 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.575019 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.576351 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.576793 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.576905 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:08.581522 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:08.581528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:08.581532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:08.581575 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:08.584354 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:08.585044 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:08.585049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:08.585051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:08.585191 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:08.585558 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:08.585563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:08.585566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:08.585880 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:08.586130 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:08.586418 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 18:38:08.586658 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 18:38:08.588717 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 2542
I20260927 18:38:08.602264 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:08.589823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.2.95.65:33189)
0927 18:38:08.590103 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:08.590109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:08.590501 (+ 392us) server_negotiation.cc:407] Connection header received
0927 18:38:08.590552 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:08.590556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:08.590620 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:08.590696 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:08.591859 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.592681 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.594072 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.594564 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.594661 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:08.596985 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:08.596990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:08.596994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:08.597027 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:08.599686 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:08.600355 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:08.600359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:08.600362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:08.600498 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:08.600834 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:08.600838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:08.600840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:08.601142 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:08.601340 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:08.601624 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 18:38:08.601871 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:38:08.603055 2665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:08.695211 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 2542
I20260927 18:38:08.710035 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 2676
Sep 27 18:38:08 dist-test-slave-9jll krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:38:08.728643 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:08.711067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.2.95.66:39913)
0927 18:38:08.711691 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:08.711699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:08.711728 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:38:08.711801 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:08.711809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:08.712090 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:08.712264 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:08.713638 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.714788 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.716135 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.716751 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.716870 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:08.721558 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:08.721573 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:08.721593 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:08.721813 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:08.724940 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:08.726060 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:08.726076 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:08.726079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:08.726264 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:08.726802 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:08.726807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:08.726809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:08.727253 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:08.727606 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:08.727893 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 18:38:08.728169 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":313,"threads_started":1}
I20260927 18:38:08.730239 2799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:08.818838 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 2676
I20260927 18:38:08.834434 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 2810
Sep 27 18:38:08 dist-test-slave-9jll krb5kdc[2436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:38:08.853138 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:08.835674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51912 (local address 127.2.95.67:35347)
0927 18:38:08.836341 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:08.836349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:08.836383 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:38:08.836454 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:08.836462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:08.836749 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:08.836904 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:08.838433 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.839756 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.841215 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:08.841932 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:08.842049 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:08.846677 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:08.846693 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:08.846714 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:08.846764 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:08.849880 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:08.850594 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:08.850607 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:08.850609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:08.850769 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:08.851295 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:08.851299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:08.851301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:08.851769 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:08.852040 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:08.852433 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 18:38:08.852712 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":330,"threads_started":1}
I20260927 18:38:08.854835 2933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:08.944015 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 2810
I20260927 18:38:08.958870 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 2449
I20260927 18:38:08.960918 2511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:09.077566 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 2449
2026-09-27T18:38:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4318 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:09 dist-test-slave-9jll krb5kdc[2960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:09 dist-test-slave-9jll krb5kdc[2960](info): set up 2 sockets
Sep 27 18:38:09 dist-test-slave-9jll krb5kdc[2960](info): commencing operation
krb5kdc: starting...
W20260927 18:38:10.104529 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:38:10 dist-test-slave-9jll krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:38:10.310567 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:37913
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:37913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:10.693711 2973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:10.694100 2973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:10.694229 2973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:10.702279 2973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:10.702419 2973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:10.702481 2973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:10.702564 2973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:10.702625 2973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:10.714020 2973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:37913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:37913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:10.715849 2973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:10.717536 2973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:10.731390 2979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:10.731415 2978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:10.731367 2981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:10.732136 2973 server_base.cc:1034] running on GCE node
I20260927 18:38:10.733100 2973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:10.734969 2973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:10.736241 2973 hybrid_clock.cc:648] HybridClock initialized: now 1790534290736186 us; error 48 us; skew 500 ppm
I20260927 18:38:10.736687 2973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:10 dist-test-slave-9jll krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534290, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:10.740449 2973 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:38:10.742393 2973 webserver.cc:460] Webserver started at http://127.2.95.126:43453/ using document root <none> and password file <none>
I20260927 18:38:10.743254 2973 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:10.743363 2973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:10.743685 2973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:10.747423 2973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "dbadf214ef19481386edef596bf38d0a"
format_stamp: "Formatted at 2026-09-27 18:38:10 on dist-test-slave-9jll"
server_key: "83c90022295d5177e50c0f5167e7f626"
server_key_iv: "a7152987fa310b92360b84eecc7dd6ee"
server_key_version: "encryptionkey@0"
I20260927 18:38:10.748291 2973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "dbadf214ef19481386edef596bf38d0a"
format_stamp: "Formatted at 2026-09-27 18:38:10 on dist-test-slave-9jll"
server_key: "83c90022295d5177e50c0f5167e7f626"
server_key_iv: "a7152987fa310b92360b84eecc7dd6ee"
server_key_version: "encryptionkey@0"
I20260927 18:38:10.754778 2973 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 18:38:10.759035 2988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:10.760859 2973 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:38:10.761075 2973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "dbadf214ef19481386edef596bf38d0a"
format_stamp: "Formatted at 2026-09-27 18:38:10 on dist-test-slave-9jll"
server_key: "83c90022295d5177e50c0f5167e7f626"
server_key_iv: "a7152987fa310b92360b84eecc7dd6ee"
server_key_version: "encryptionkey@0"
I20260927 18:38:10.761307 2973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:10.778748 2973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:10.783897 2973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:10.784178 2973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:10.803960 2973 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:37913
I20260927 18:38:10.803987 3039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:37913 every 8 connection(s)
I20260927 18:38:10.805687 2973 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:10.810491 3040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:10.814538 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 2973
I20260927 18:38:10.814795 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:10.815653 2429 external_mini_cluster.cc:1428] Setting key a9e32a0803777b5dcf26257b4dcddc0c
I20260927 18:38:10.826148 3040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a: Bootstrap starting.
I20260927 18:38:10.830971 3040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:10.832749 3040 log.cc:824] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:38:10 dist-test-slave-9jll krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:10.836760 3040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a: No bootstrap required, opened a new log
I20260927 18:38:10.843406 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:10.818326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.2.95.126:37913)
0927 18:38:10.819262 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:10.819275 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:10.819325 (+ 50us) server_negotiation.cc:407] Connection header received
0927 18:38:10.821238 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:10.821270 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:10.821871 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:10.822307 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:10.824279 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:10.825947 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:10.827768 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:10.828582 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:10.828744 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:10.834260 (+ 5516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:10.834272 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:10.834298 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:10.834354 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:10.838188 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:10.839085 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:10.839094 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:10.839103 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:10.839328 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:10.839763 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:10.839768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:10.839770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:10.840494 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:10.840837 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:10.841449 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 18:38:10.842053 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":430,"threads_started":1}
I20260927 18:38:10.845705 3040 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbadf214ef19481386edef596bf38d0a" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37913 } }
I20260927 18:38:10.846153 3040 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:10.846256 3040 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbadf214ef19481386edef596bf38d0a, State: Initialized, Role: FOLLOWER
I20260927 18:38:10.846963 3040 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbadf214ef19481386edef596bf38d0a" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37913 } }
I20260927 18:38:10.847249 3040 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:10.847375 3040 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:10.847539 3040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:10.850050 3040 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbadf214ef19481386edef596bf38d0a" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37913 } }
I20260927 18:38:10.850642 3040 leader_election.cc:302] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbadf214ef19481386edef596bf38d0a; no voters:
I20260927 18:38:10.851342 3040 leader_election.cc:288] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:10.851475 3045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:10.853879 3040 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:10.854051 3045 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [term 1 LEADER]: Becoming Leader. State: Replica: dbadf214ef19481386edef596bf38d0a, State: Running, Role: LEADER
I20260927 18:38:10.854632 3045 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbadf214ef19481386edef596bf38d0a" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37913 } }
I20260927 18:38:10.858247 3047 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbadf214ef19481386edef596bf38d0a. Latest consensus state: current_term: 1 leader_uuid: "dbadf214ef19481386edef596bf38d0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbadf214ef19481386edef596bf38d0a" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37913 } } }
I20260927 18:38:10.858319 3046 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbadf214ef19481386edef596bf38d0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbadf214ef19481386edef596bf38d0a" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37913 } } }
I20260927 18:38:10.858842 3046 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:10.858839 3047 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:10.861967 3055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:10.868747 3055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:10.879787 3055 catalog_manager.cc:1347] Generated new cluster ID: 2edea926d03142449bb36703672cd77f
I20260927 18:38:10.879935 3055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:10.902760 3055 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:10.904735 3055 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:10.916379 3055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbadf214ef19481386edef596bf38d0a: Generated new TSK 0
I20260927 18:38:10.917850 3055 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:38:10.979311 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37913
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:11.375835 3066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:11.376211 3066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:11.376313 3066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:11.384398 3066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:11.384567 3066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:11.384704 3066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:38:11.395903 3066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:11.397622 3066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:11.399271 3066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:11.414335 3071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:11.414362 3072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:11.414335 3074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:11.415733 3066 server_base.cc:1034] running on GCE node
I20260927 18:38:11.416484 3066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:11.417694 3066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:11.418941 3066 hybrid_clock.cc:648] HybridClock initialized: now 1790534291418885 us; error 75 us; skew 500 ppm
I20260927 18:38:11.419301 3066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:11 dist-test-slave-9jll krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534291, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:11.422921 3066 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:38:11.424628 3066 webserver.cc:460] Webserver started at http://127.2.95.65:45505/ using document root <none> and password file <none>
I20260927 18:38:11.425448 3066 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:11.425602 3066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:11.425983 3066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:11.429450 3066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "07d6ed1bfaf14f51a9b444a2d995197b"
format_stamp: "Formatted at 2026-09-27 18:38:11 on dist-test-slave-9jll"
server_key: "ff23d65d058e6f2714a6907ab1fd57a7"
server_key_iv: "8c722b23a4047c39dea8fc5bc7f3899f"
server_key_version: "encryptionkey@0"
I20260927 18:38:11.430447 3066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "07d6ed1bfaf14f51a9b444a2d995197b"
format_stamp: "Formatted at 2026-09-27 18:38:11 on dist-test-slave-9jll"
server_key: "ff23d65d058e6f2714a6907ab1fd57a7"
server_key_iv: "8c722b23a4047c39dea8fc5bc7f3899f"
server_key_version: "encryptionkey@0"
I20260927 18:38:11.436865 3066 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 18:38:11.441184 3081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:11.442967 3066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:38:11.443156 3066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "07d6ed1bfaf14f51a9b444a2d995197b"
format_stamp: "Formatted at 2026-09-27 18:38:11 on dist-test-slave-9jll"
server_key: "ff23d65d058e6f2714a6907ab1fd57a7"
server_key_iv: "8c722b23a4047c39dea8fc5bc7f3899f"
server_key_version: "encryptionkey@0"
I20260927 18:38:11.443379 3066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:11.463410 3066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:11.469197 3066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:11.469537 3066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:11.470813 3066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:11.472445 3066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:11.472543 3066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:11.472673 3066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:11.472725 3066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:11.507784 3066 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:42553
I20260927 18:38:11.507853 3193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:42553 every 8 connection(s)
I20260927 18:38:11.509294 3066 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:11.511950 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 3066
I20260927 18:38:11.512230 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:11.513255 2429 external_mini_cluster.cc:1428] Setting key d509fc772fa4450d3e8cba509bd77d8d
Sep 27 18:38:11 dist-test-slave-9jll krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534291, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
I20260927 18:38:11.535310 3196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:11.512501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:37691 (local address 127.2.95.126:37913)
0927 18:38:11.513008 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:11.513015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:11.514574 (+ 1559us) server_negotiation.cc:407] Connection header received
0927 18:38:11.517038 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:11.517044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:11.517155 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:11.517260 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:11.519520 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:11.520584 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:11.522617 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:11.523070 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:11.523182 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:11.528228 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:11.528236 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:11.528240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:11.528289 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:11.531517 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:11.532520 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:11.532527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:11.532530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:11.532690 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:11.533191 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:11.533196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:11.533198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:11.533539 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:11.533732 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:11.534606 (+ 874us) server_negotiation.cc:299] Negotiation successful
0927 18:38:11.534894 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":256,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
I20260927 18:38:11.536556 3197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:11.513345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37913 (local address 127.2.95.65:37691)
0927 18:38:11.514386 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:11.514444 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:11.516715 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:11.517587 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:11.517599 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:11.518342 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:11.519280 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:11.519305 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:11.520752 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:11.520757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:11.522453 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:11.522468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:11.523332 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:11.523340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:11.523716 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:11.524603 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:11.524643 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:11.527849 (+ 3206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:11.531736 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:11.531747 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:11.531769 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:11.532348 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:11.532819 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:11.532824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:11.532828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:11.533056 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:11.533710 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:11.533721 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:11.534180 (+ 459us) client_negotiation.cc:769] Sending connection context
0927 18:38:11.534465 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 18:38:11.535153 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":429,"threads_started":1}
I20260927 18:38:11.538888 3194 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37913
I20260927 18:38:11.539256 3194 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:11.540184 3194 heartbeater.cc:511] Master 127.2.95.126:37913 requested a full tablet report, sending...
I20260927 18:38:11.543061 3005 ts_manager.cc:194] Registered new tserver with Master: 07d6ed1bfaf14f51a9b444a2d995197b (127.2.95.65:42553)
I20260927 18:38:11.544804 3005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:37691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:38:11.565204 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37913
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:11.952023 3200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:11.952404 3200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:11.952548 3200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:11.960932 3200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:11.961122 3200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:11.961303 3200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:38:11.973013 3200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:11.974961 3200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:11.976727 3200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:11.992008 3208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:11.992105 3206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:11.992077 3205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:11.992630 3200 server_base.cc:1034] running on GCE node
I20260927 18:38:11.993371 3200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:11.994524 3200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:11.995744 3200 hybrid_clock.cc:648] HybridClock initialized: now 1790534291995684 us; error 55 us; skew 500 ppm
I20260927 18:38:11.996102 3200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:11 dist-test-slave-9jll krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534291, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:11.999732 3200 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:38:12.001593 3200 webserver.cc:460] Webserver started at http://127.2.95.66:34197/ using document root <none> and password file <none>
I20260927 18:38:12.002492 3200 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:12.002606 3200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:12.002942 3200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:12.006644 3200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "dab4ffbae3e54251a14c6fab3e628773"
format_stamp: "Formatted at 2026-09-27 18:38:11 on dist-test-slave-9jll"
server_key: "28eb06b75d43cdd0fa68664901cec9bd"
server_key_iv: "f365dda098bf0bf54542a39e5d4d68a9"
server_key_version: "encryptionkey@0"
I20260927 18:38:12.007865 3200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "dab4ffbae3e54251a14c6fab3e628773"
format_stamp: "Formatted at 2026-09-27 18:38:11 on dist-test-slave-9jll"
server_key: "28eb06b75d43cdd0fa68664901cec9bd"
server_key_iv: "f365dda098bf0bf54542a39e5d4d68a9"
server_key_version: "encryptionkey@0"
I20260927 18:38:12.015234 3200 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 18:38:12.019816 3215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:12.021597 3200 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 18:38:12.021839 3200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "dab4ffbae3e54251a14c6fab3e628773"
format_stamp: "Formatted at 2026-09-27 18:38:11 on dist-test-slave-9jll"
server_key: "28eb06b75d43cdd0fa68664901cec9bd"
server_key_iv: "f365dda098bf0bf54542a39e5d4d68a9"
server_key_version: "encryptionkey@0"
I20260927 18:38:12.022084 3200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:12.049224 3200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:12.054616 3200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:12.054911 3200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:12.056116 3200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:12.057889 3200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:12.057971 3200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:12.058073 3200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:12.058117 3200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:12.092751 3200 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:38509
I20260927 18:38:12.092805 3327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:38509 every 8 connection(s)
I20260927 18:38:12.094285 3200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:12.097597 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 3200
I20260927 18:38:12.097851 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:12.098361 2429 external_mini_cluster.cc:1428] Setting key 02c12c9d7769e7fad0424c632be4e397
Sep 27 18:38:12 dist-test-slave-9jll krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534291, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
I20260927 18:38:12.118865 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.097338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:47363 (local address 127.2.95.126:37913)
0927 18:38:12.097920 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:12.097928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:12.099173 (+ 1245us) server_negotiation.cc:407] Connection header received
0927 18:38:12.101737 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:12.101744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:12.101846 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:12.101975 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:12.103850 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.104983 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.106854 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.107306 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.107400 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:12.111932 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:12.111938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:12.111942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:12.111991 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:12.114801 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:12.115853 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:12.115858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:12.115860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:12.116024 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:12.116620 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:12.116624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:12.116627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:12.116965 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:12.117115 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:12.118303 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0927 18:38:12.118531 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
I20260927 18:38:12.120179 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.097979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37913 (local address 127.2.95.66:47363)
0927 18:38:12.098946 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:12.099000 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:12.101396 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:12.102189 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:12.102199 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:12.102814 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:12.103589 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:12.103608 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.105132 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.105135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:12.106704 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:12.106719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.107530 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.107536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:12.107840 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:12.108627 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:12.108657 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:12.111642 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:12.114973 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:12.114986 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:12.115012 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:12.115688 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:12.116142 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:12.116147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:12.116151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:12.116446 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:12.117118 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:12.117128 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:12.117736 (+ 608us) client_negotiation.cc:769] Sending connection context
0927 18:38:12.118119 (+ 383us) client_negotiation.cc:240] Negotiation successful
0927 18:38:12.118684 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":356,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
I20260927 18:38:12.122170 3328 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37913
I20260927 18:38:12.122555 3328 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:12.123441 3328 heartbeater.cc:511] Master 127.2.95.126:37913 requested a full tablet report, sending...
I20260927 18:38:12.125499 3005 ts_manager.cc:194] Registered new tserver with Master: dab4ffbae3e54251a14c6fab3e628773 (127.2.95.66:38509)
I20260927 18:38:12.126552 3005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:47363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:38:12.146055 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37913
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:38:12.548606 3194 heartbeater.cc:503] Master 127.2.95.126:37913 was elected leader, sending a full tablet report...
W20260927 18:38:12.552970 3334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:12.553311 3334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:12.553426 3334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:12.561380 3334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:12.561513 3334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:12.561653 3334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:38:12.572990 3334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:12.574767 3334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:12.576407 3334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:12.591619 3340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:12.591639 3342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:12.591527 3339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:12.592429 3334 server_base.cc:1034] running on GCE node
I20260927 18:38:12.593202 3334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:12.594452 3334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:12.595674 3334 hybrid_clock.cc:648] HybridClock initialized: now 1790534292595606 us; error 60 us; skew 500 ppm
I20260927 18:38:12.596071 3334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:12 dist-test-slave-9jll krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534292, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:12.599707 3334 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:38:12.601349 3334 webserver.cc:460] Webserver started at http://127.2.95.67:45547/ using document root <none> and password file <none>
I20260927 18:38:12.602250 3334 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:12.602357 3334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:12.602658 3334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:12.606118 3334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "64360a4cd45d4334b4f401180523949c"
format_stamp: "Formatted at 2026-09-27 18:38:12 on dist-test-slave-9jll"
server_key: "1d2ece7bc16e157f32b81418615c6274"
server_key_iv: "2895e8e69cf03a4a5146f35564d9fabb"
server_key_version: "encryptionkey@0"
I20260927 18:38:12.606971 3334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "64360a4cd45d4334b4f401180523949c"
format_stamp: "Formatted at 2026-09-27 18:38:12 on dist-test-slave-9jll"
server_key: "1d2ece7bc16e157f32b81418615c6274"
server_key_iv: "2895e8e69cf03a4a5146f35564d9fabb"
server_key_version: "encryptionkey@0"
I20260927 18:38:12.613211 3334 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:38:12.617535 3349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:12.619549 3334 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:38:12.619752 3334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "64360a4cd45d4334b4f401180523949c"
format_stamp: "Formatted at 2026-09-27 18:38:12 on dist-test-slave-9jll"
server_key: "1d2ece7bc16e157f32b81418615c6274"
server_key_iv: "2895e8e69cf03a4a5146f35564d9fabb"
server_key_version: "encryptionkey@0"
I20260927 18:38:12.620002 3334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:12.650426 3334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:12.656363 3334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:12.656728 3334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:12.658047 3334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:12.659689 3334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:12.659777 3334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:12.659914 3334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:12.659986 3334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:12.695591 3334 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:37263
I20260927 18:38:12.695606 3461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:37263 every 8 connection(s)
I20260927 18:38:12.697217 3334 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:38:12.698870 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 3334
I20260927 18:38:12.699110 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:12.699627 2429 external_mini_cluster.cc:1428] Setting key 3704e451eb443f5518923e324b76485e
Sep 27 18:38:12 dist-test-slave-9jll krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534292, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:12.722324 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.700746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:55051 (local address 127.2.95.126:37913)
0927 18:38:12.701527 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:12.701535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:12.703203 (+ 1668us) server_negotiation.cc:407] Connection header received
0927 18:38:12.705416 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:12.705420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:12.705525 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:12.705658 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:12.707584 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.708456 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.710420 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.710894 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.711004 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:12.715598 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:12.715603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:12.715607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:12.715654 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:12.718870 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:12.719831 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:12.719835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:12.719838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:12.719988 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:12.720396 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:12.720400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:12.720402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:12.720717 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:12.720842 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:12.721752 (+ 910us) server_negotiation.cc:299] Negotiation successful
0927 18:38:12.722009 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":459,"threads_started":1}
I20260927 18:38:12.723457 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.701542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37913 (local address 127.2.95.67:55051)
0927 18:38:12.703005 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:12.703072 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:12.705129 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:12.705970 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:12.705981 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:12.706688 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:12.707393 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:12.707413 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.708651 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.708656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:12.710257 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:12.710273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.711195 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.711202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:12.711460 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:12.712348 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:12.712376 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:12.715327 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:12.719046 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:12.719056 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:12.719076 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:12.719685 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:12.720080 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:12.720084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:12.720086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:12.720298 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:12.720838 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:12.720845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:12.721270 (+ 425us) client_negotiation.cc:769] Sending connection context
0927 18:38:12.721574 (+ 304us) client_negotiation.cc:240] Negotiation successful
0927 18:38:12.722157 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"thread_start_us":790,"threads_started":1}
I20260927 18:38:12.725148 3462 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37913
I20260927 18:38:12.725551 3462 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:12.726471 3462 heartbeater.cc:511] Master 127.2.95.126:37913 requested a full tablet report, sending...
I20260927 18:38:12.728262 3005 ts_manager.cc:194] Registered new tserver with Master: 64360a4cd45d4334b4f401180523949c (127.2.95.67:37263)
I20260927 18:38:12.729344 3005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:55051
I20260927 18:38:12.729400 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:38:12.752226 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.737467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.2.95.126:37913)
0927 18:38:12.737744 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:12.737751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:12.738134 (+ 383us) server_negotiation.cc:407] Connection header received
0927 18:38:12.738283 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:12.738293 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:12.738386 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:12.738480 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:12.740186 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.740991 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.742636 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.743173 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.743301 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:12.746066 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:12.746071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:12.746075 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:12.746119 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:12.749088 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:12.749982 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:12.749986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:12.749990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:12.750139 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:12.750629 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:12.750634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:12.750636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:12.751088 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:12.751310 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:12.751698 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 18:38:12.751904 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 18:38:12.760443 3005 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:38:12.768314 3005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:12.796149 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.782620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:38509 (local address 127.0.0.1:42710)
0927 18:38:12.783408 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:12.783447 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:12.783671 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:12.787165 (+ 3494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:12.787173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:12.787216 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:12.787707 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:12.787722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.790128 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.790134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:12.792743 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:12.792763 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.795138 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.795144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:12.795324 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:12.795376 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 18:38:12.795465 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:38:12.795659 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":356,"threads_started":1}
I20260927 18:38:12.796875 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.782876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.2.95.66:38509)
0927 18:38:12.784665 (+ 1789us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:12.784676 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:12.784707 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:38:12.784818 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:12.784826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:12.786913 (+ 2087us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:12.787926 (+ 1013us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:12.788183 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.789961 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.792992 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.794925 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.796004 (+ 1079us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:12.796276 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:12.796357 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 18:38:12.796557 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1596,"thread_start_us":1499,"threads_started":1}
I20260927 18:38:12.802042 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.783462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.2.95.67:37263)
0927 18:38:12.794890 (+ 11428us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:12.794901 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:12.794944 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:38:12.795075 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:12.795086 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:12.795410 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:12.795646 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:12.796407 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.797897 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.799975 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.800943 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.801059 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:12.801321 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:12.801496 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 18:38:12.801694 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11223,"thread_start_us":11066,"threads_started":1}
I20260927 18:38:12.802044 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.782734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:37263 (local address 127.0.0.1:57678)
0927 18:38:12.786321 (+ 3587us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:12.786342 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:12.786474 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:12.795637 (+ 9163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:12.795641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:12.795668 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:12.796149 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:12.796160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.798043 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.798060 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:12.799834 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:12.799846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.801142 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.801149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:12.801321 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:12.801357 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:38:12.801443 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 18:38:12.801594 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3449,"mutex_wait_us":1,"thread_start_us":3298,"threads_started":1}
I20260927 18:38:12.801992 3263 tablet_service.cc:1467] Processing CreateTablet for tablet a492f6e33e914a82b6686f5c7a0ab800 (DEFAULT_TABLE table=test-table [id=dc28e79693b9488ebb0c9bce80ec9458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:12.803261 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.784454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:42553 (local address 127.0.0.1:42004)
0927 18:38:12.787639 (+ 3185us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:12.787658 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:12.787792 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:12.795911 (+ 8119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:12.795915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:12.795941 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:12.796433 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:12.796444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.798580 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.798585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:12.800698 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:12.800712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.802467 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.802473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:12.802633 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:12.802666 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:38:12.802753 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:38:12.802887 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3047,"thread_start_us":3247,"threads_started":1}
I20260927 18:38:12.803390 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:12.784969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.2.95.65:42553)
0927 18:38:12.795300 (+ 10331us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:12.795307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:12.795335 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:38:12.795432 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:12.795439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:12.795681 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:12.796621 (+ 940us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:12.796861 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.798412 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.800902 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:12.802294 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:12.802407 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:12.802639 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:12.802887 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 18:38:12.803071 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10155,"thread_start_us":10004,"threads_started":1}
I20260927 18:38:12.804486 3263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a492f6e33e914a82b6686f5c7a0ab800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:12.807691 3129 tablet_service.cc:1467] Processing CreateTablet for tablet a492f6e33e914a82b6686f5c7a0ab800 (DEFAULT_TABLE table=test-table [id=dc28e79693b9488ebb0c9bce80ec9458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:12.808080 3397 tablet_service.cc:1467] Processing CreateTablet for tablet a492f6e33e914a82b6686f5c7a0ab800 (DEFAULT_TABLE table=test-table [id=dc28e79693b9488ebb0c9bce80ec9458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:12.809486 3397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a492f6e33e914a82b6686f5c7a0ab800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:12.809477 3129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a492f6e33e914a82b6686f5c7a0ab800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:12.830629 3482 tablet_bootstrap.cc:492] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b: Bootstrap starting.
I20260927 18:38:12.830601 3481 tablet_bootstrap.cc:492] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773: Bootstrap starting.
I20260927 18:38:12.831599 3483 tablet_bootstrap.cc:492] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c: Bootstrap starting.
I20260927 18:38:12.835157 3481 tablet_bootstrap.cc:654] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:12.835381 3483 tablet_bootstrap.cc:654] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:12.835623 3482 tablet_bootstrap.cc:654] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:12.836654 3483 log.cc:824] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:12.836652 3481 log.cc:824] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:12.836825 3482 log.cc:824] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:12.840482 3482 tablet_bootstrap.cc:492] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b: No bootstrap required, opened a new log
I20260927 18:38:12.840493 3483 tablet_bootstrap.cc:492] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c: No bootstrap required, opened a new log
I20260927 18:38:12.840492 3481 tablet_bootstrap.cc:492] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773: No bootstrap required, opened a new log
I20260927 18:38:12.840826 3481 ts_tablet_manager.cc:1397] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 18:38:12.840826 3483 ts_tablet_manager.cc:1397] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 18:38:12.840821 3482 ts_tablet_manager.cc:1397] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260927 18:38:12.849548 3482 raft_consensus.cc:348] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab4ffbae3e54251a14c6fab3e628773" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38509 } } peers { permanent_uuid: "64360a4cd45d4334b4f401180523949c" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37263 } } peers { permanent_uuid: "07d6ed1bfaf14f51a9b444a2d995197b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42553 } }
I20260927 18:38:12.850143 3482 raft_consensus.cc:374] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:12.850270 3482 raft_consensus.cc:729] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d6ed1bfaf14f51a9b444a2d995197b, State: Initialized, Role: FOLLOWER
I20260927 18:38:12.850991 3482 consensus_queue.cc:260] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab4ffbae3e54251a14c6fab3e628773" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38509 } } peers { permanent_uuid: "64360a4cd45d4334b4f401180523949c" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37263 } } peers { permanent_uuid: "07d6ed1bfaf14f51a9b444a2d995197b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42553 } }
I20260927 18:38:12.852219 3481 raft_consensus.cc:348] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab4ffbae3e54251a14c6fab3e628773" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38509 } } peers { permanent_uuid: "64360a4cd45d4334b4f401180523949c" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37263 } } peers { permanent_uuid: "07d6ed1bfaf14f51a9b444a2d995197b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42553 } }
I20260927 18:38:12.852218 3483 raft_consensus.cc:348] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab4ffbae3e54251a14c6fab3e628773" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38509 } } peers { permanent_uuid: "64360a4cd45d4334b4f401180523949c" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37263 } } peers { permanent_uuid: "07d6ed1bfaf14f51a9b444a2d995197b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42553 } }
I20260927 18:38:12.852705 3483 raft_consensus.cc:374] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:12.852788 3481 raft_consensus.cc:374] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:12.852849 3483 raft_consensus.cc:729] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64360a4cd45d4334b4f401180523949c, State: Initialized, Role: FOLLOWER
I20260927 18:38:12.852972 3481 raft_consensus.cc:729] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab4ffbae3e54251a14c6fab3e628773, State: Initialized, Role: FOLLOWER
I20260927 18:38:12.853101 3482 ts_tablet_manager.cc:1428] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260927 18:38:12.853632 3483 consensus_queue.cc:260] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab4ffbae3e54251a14c6fab3e628773" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38509 } } peers { permanent_uuid: "64360a4cd45d4334b4f401180523949c" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37263 } } peers { permanent_uuid: "07d6ed1bfaf14f51a9b444a2d995197b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42553 } }
I20260927 18:38:12.853632 3481 consensus_queue.cc:260] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab4ffbae3e54251a14c6fab3e628773" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38509 } } peers { permanent_uuid: "64360a4cd45d4334b4f401180523949c" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37263 } } peers { permanent_uuid: "07d6ed1bfaf14f51a9b444a2d995197b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42553 } }
I20260927 18:38:12.855392 3462 heartbeater.cc:503] Master 127.2.95.126:37913 was elected leader, sending a full tablet report...
I20260927 18:38:12.855618 3481 ts_tablet_manager.cc:1428] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260927 18:38:12.855964 3483 ts_tablet_manager.cc:1428] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260927 18:38:12.856375 3328 heartbeater.cc:503] Master 127.2.95.126:37913 was elected leader, sending a full tablet report...
W20260927 18:38:12.949254 3463 tablet.cc:2369] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:38:13.012302 3195 tablet.cc:2369] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:38:13.030508 3489 raft_consensus.cc:482] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:13.030874 3489 raft_consensus.cc:504] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab4ffbae3e54251a14c6fab3e628773" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38509 } } peers { permanent_uuid: "64360a4cd45d4334b4f401180523949c" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37263 } } peers { permanent_uuid: "07d6ed1bfaf14f51a9b444a2d995197b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42553 } }
I20260927 18:38:13.032912 3489 leader_election.cc:288] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dab4ffbae3e54251a14c6fab3e628773 (127.2.95.66:38509), 07d6ed1bfaf14f51a9b444a2d995197b (127.2.95.65:42553)
I20260927 18:38:13.040436 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:13.033525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:47677 (local address 127.2.95.66:38509)
0927 18:38:13.033798 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:13.033806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:13.033833 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:38:13.034037 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:13.034044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:13.034137 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:13.034261 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:13.034792 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.035837 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.038143 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.039447 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.039570 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:13.039656 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:13.040071 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 18:38:13.040209 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:38:13.040504 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:13.033392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:38509 (local address 127.2.95.67:47677)
0927 18:38:13.033637 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:13.033661 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:13.033861 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:13.034310 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:13.034317 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:13.034365 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:13.034624 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:13.034638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.035997 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.036003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:13.037959 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:13.037976 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.039636 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.039643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:13.039849 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:13.039901 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 18:38:13.040005 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:38:13.040160 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":26}
I20260927 18:38:13.041807 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:13.034307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:55751 (local address 127.2.95.65:42553)
0927 18:38:13.034515 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:13.034521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:13.034933 (+ 412us) server_negotiation.cc:407] Connection header received
0927 18:38:13.035159 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:13.035165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:13.035236 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:13.035318 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:13.035907 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.037008 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.039564 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.040786 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.040892 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:13.040973 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:13.041409 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 18:38:13.041531 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:38:13.041850 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:13.034018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:42553 (local address 127.2.95.67:55751)
0927 18:38:13.034821 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:13.034841 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:13.034976 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:13.035351 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:13.035355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:13.035383 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:13.035739 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:13.035750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.037206 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.037211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:13.039353 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:13.039369 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.041036 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.041041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:13.041226 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:13.041271 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 18:38:13.041370 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:38:13.041511 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":43,"thread_start_us":511,"threads_started":1}
I20260927 18:38:13.042222 3283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a492f6e33e914a82b6686f5c7a0ab800" candidate_uuid: "64360a4cd45d4334b4f401180523949c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab4ffbae3e54251a14c6fab3e628773" is_pre_election: true
I20260927 18:38:13.042917 3283 raft_consensus.cc:2454] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64360a4cd45d4334b4f401180523949c in term 0.
I20260927 18:38:13.042881 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a492f6e33e914a82b6686f5c7a0ab800" candidate_uuid: "64360a4cd45d4334b4f401180523949c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d6ed1bfaf14f51a9b444a2d995197b" is_pre_election: true
I20260927 18:38:13.043445 3149 raft_consensus.cc:2454] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64360a4cd45d4334b4f401180523949c in term 0.
I20260927 18:38:13.043684 3351 leader_election.cc:302] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64360a4cd45d4334b4f401180523949c, dab4ffbae3e54251a14c6fab3e628773; no voters:
I20260927 18:38:13.044097 3489 raft_consensus.cc:2781] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:13.044226 3489 raft_consensus.cc:482] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:13.044332 3489 raft_consensus.cc:3037] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:13.046947 3489 raft_consensus.cc:504] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab4ffbae3e54251a14c6fab3e628773" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38509 } } peers { permanent_uuid: "64360a4cd45d4334b4f401180523949c" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37263 } } peers { permanent_uuid: "07d6ed1bfaf14f51a9b444a2d995197b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42553 } }
I20260927 18:38:13.047619 3489 leader_election.cc:288] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [CANDIDATE]: Term 1 election: Requested vote from peers dab4ffbae3e54251a14c6fab3e628773 (127.2.95.66:38509), 07d6ed1bfaf14f51a9b444a2d995197b (127.2.95.65:42553)
I20260927 18:38:13.048235 3283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a492f6e33e914a82b6686f5c7a0ab800" candidate_uuid: "64360a4cd45d4334b4f401180523949c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab4ffbae3e54251a14c6fab3e628773"
I20260927 18:38:13.048264 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a492f6e33e914a82b6686f5c7a0ab800" candidate_uuid: "64360a4cd45d4334b4f401180523949c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07d6ed1bfaf14f51a9b444a2d995197b"
I20260927 18:38:13.048470 3283 raft_consensus.cc:3037] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:13.048478 3149 raft_consensus.cc:3037] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:13.050978 3149 raft_consensus.cc:2454] T a492f6e33e914a82b6686f5c7a0ab800 P 07d6ed1bfaf14f51a9b444a2d995197b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64360a4cd45d4334b4f401180523949c in term 1.
I20260927 18:38:13.050974 3283 raft_consensus.cc:2454] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64360a4cd45d4334b4f401180523949c in term 1.
I20260927 18:38:13.051395 3351 leader_election.cc:302] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64360a4cd45d4334b4f401180523949c, dab4ffbae3e54251a14c6fab3e628773; no voters:
I20260927 18:38:13.051718 3489 raft_consensus.cc:2781] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:13.052220 3489 raft_consensus.cc:686] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [term 1 LEADER]: Becoming Leader. State: Replica: 64360a4cd45d4334b4f401180523949c, State: Running, Role: LEADER
I20260927 18:38:13.052800 3489 consensus_queue.cc:237] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab4ffbae3e54251a14c6fab3e628773" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38509 } } peers { permanent_uuid: "64360a4cd45d4334b4f401180523949c" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37263 } } peers { permanent_uuid: "07d6ed1bfaf14f51a9b444a2d995197b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42553 } }
I20260927 18:38:13.058847 3004 catalog_manager.cc:5495] T a492f6e33e914a82b6686f5c7a0ab800 P 64360a4cd45d4334b4f401180523949c reported cstate change: term changed from 0 to 1, leader changed from <none> to 64360a4cd45d4334b4f401180523949c (127.2.95.67). New cstate: current_term: 1 leader_uuid: "64360a4cd45d4334b4f401180523949c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab4ffbae3e54251a14c6fab3e628773" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64360a4cd45d4334b4f401180523949c" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07d6ed1bfaf14f51a9b444a2d995197b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42553 } health_report { overall_health: UNKNOWN } } }
W20260927 18:38:13.096633 3329 tablet.cc:2369] T a492f6e33e914a82b6686f5c7a0ab800 P dab4ffbae3e54251a14c6fab3e628773: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 18:38:13 dist-test-slave-9jll krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:13.117400 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:13.099426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.2.95.65:42553)
0927 18:38:13.099661 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:13.099669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:13.100312 (+ 643us) server_negotiation.cc:407] Connection header received
0927 18:38:13.100562 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:13.100567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:13.100651 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:13.100747 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:13.103028 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.103846 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.105656 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.106257 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.106379 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:13.111070 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:13.111096 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:13.111115 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:13.111169 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:13.114307 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.115057 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.115065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.115068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.115231 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.115620 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.115631 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.115634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.116062 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:13.116371 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:13.116805 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 18:38:13.116999 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 18:38:13 dist-test-slave-9jll krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:38:13 dist-test-slave-9jll krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:13.168263 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:13.152849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.2.95.65:42553)
0927 18:38:13.153057 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:13.153065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:13.153482 (+ 417us) server_negotiation.cc:407] Connection header received
0927 18:38:13.153730 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:13.153737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:13.153820 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:13.153917 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:13.155426 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.156218 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.157562 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.158070 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.158205 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:13.162358 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:13.162364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:13.162368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:13.162409 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:13.165341 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.166166 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.166171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.166173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.166330 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.166789 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.166793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.166795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.167110 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:13.167376 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:13.167666 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 18:38:13.167916 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 18:38:13 dist-test-slave-9jll krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:38:13 dist-test-slave-9jll krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:13.216269 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:13.201830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.2.95.65:42553)
0927 18:38:13.202119 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:13.202126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:13.202318 (+ 192us) server_negotiation.cc:407] Connection header received
0927 18:38:13.202500 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:13.202505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:13.202601 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:13.202697 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:13.204304 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.205111 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.206455 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.206897 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.206992 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:13.211153 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:13.211160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:13.211163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:13.211213 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:13.213883 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.214553 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.214558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.214560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.214682 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.214953 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.214957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.214959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.215261 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:13.215467 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:13.215721 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 18:38:13.215916 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 18:38:13.218482 3109 tablet_service.cc:2908] Rejecting scan request for tablet a492f6e33e914a82b6686f5c7a0ab800: Uninitialized: clean time has not yet been initialized
I20260927 18:38:13.222528 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 3066
I20260927 18:38:13.236001 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:13.223602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.2.95.65:42553)
0927 18:38:13.223804 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:13.223810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:13.224245 (+ 435us) server_negotiation.cc:407] Connection header received
0927 18:38:13.224411 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:13.224415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:13.224496 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:13.224589 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:13.225968 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.226876 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.228127 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.228568 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.228676 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:13.230955 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:13.230960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:13.230964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:13.230996 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:13.233449 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.234133 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.234139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.234141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.234285 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.234714 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.234718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.234720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.235092 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:13.235325 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:13.235572 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 18:38:13.235744 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:38:13.236713 3189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:13.333139 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 3066
I20260927 18:38:13.348515 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 3200
Sep 27 18:38:13 dist-test-slave-9jll krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:38:13.364930 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:13.349577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.2.95.66:38509)
0927 18:38:13.349819 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:13.349825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:13.349848 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:38:13.349960 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:13.349965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:13.350040 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:13.350136 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:13.351489 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.352308 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.353769 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.354231 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.354339 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:13.358927 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:13.358945 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:13.358966 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:13.359015 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:13.362072 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.362798 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.362806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.362809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.362956 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.363343 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.363346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.363348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.363787 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:13.364031 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:13.364361 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 18:38:13.364568 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 18:38:13.365649 3323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:13.458082 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 3200
I20260927 18:38:13.473028 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 3334
Sep 27 18:38:13 dist-test-slave-9jll krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:38:13.489336 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:13.474024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.2.95.67:37263)
0927 18:38:13.474553 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:13.474562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:13.474580 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:38:13.474645 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:13.474649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:13.474715 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:13.474820 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:13.476233 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.477003 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.478490 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:13.478953 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:13.479073 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:13.483578 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:13.483595 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:13.483618 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:13.483676 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:13.486764 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.487439 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.487447 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.487450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.487612 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:13.487904 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:13.487906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:13.487908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:13.488284 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:13.488514 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:13.488738 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 18:38:13.488918 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":296,"threads_started":1}
I20260927 18:38:13.490162 3457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:13.584542 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 3334
I20260927 18:38:13.599470 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 2973
I20260927 18:38:13.600796 3035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:13.721446 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 2973
2026-09-27T18:38:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4643 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:13 dist-test-slave-9jll krb5kdc[3511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:13 dist-test-slave-9jll krb5kdc[3511](info): set up 2 sockets
Sep 27 18:38:13 dist-test-slave-9jll krb5kdc[3511](info): commencing operation
krb5kdc: starting...
W20260927 18:38:14.735410 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:38:14 dist-test-slave-9jll krb5kdc[3511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:38:14.942416 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:45651
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:15.316497 3524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:15.316901 3524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:15.317055 3524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:15.325230 3524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:15.325385 3524 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:38:15.325443 3524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:15.325511 3524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:15.325594 3524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:15.325718 3524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:15.337095 3524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:45651
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:15.338876 3524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:15.340714 3524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:15.354801 3529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:15.354729 3532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:15.354756 3530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:15.355579 3524 server_base.cc:1034] running on GCE node
I20260927 18:38:15.356566 3524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:15.358327 3524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:15.359598 3524 hybrid_clock.cc:648] HybridClock initialized: now 1790534295359516 us; error 51 us; skew 500 ppm
I20260927 18:38:15.360013 3524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:15 dist-test-slave-9jll krb5kdc[3511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:15.363795 3524 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:38:15.365756 3524 webserver.cc:460] Webserver started at http://127.2.95.126:46577/ using document root <none> and password file <none>
I20260927 18:38:15.366669 3524 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:15.366793 3524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:15.367168 3524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:15.371044 3524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "825a34c3048c4f8393fe7fbbde109883"
format_stamp: "Formatted at 2026-09-27 18:38:15 on dist-test-slave-9jll"
server_key: "6f3a16f41b7a523f0f1a2cde1953528f"
server_key_iv: "d0a25c8721f9da0947be6a8f089815c3"
server_key_version: "encryptionkey@0"
I20260927 18:38:15.372021 3524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "825a34c3048c4f8393fe7fbbde109883"
format_stamp: "Formatted at 2026-09-27 18:38:15 on dist-test-slave-9jll"
server_key: "6f3a16f41b7a523f0f1a2cde1953528f"
server_key_iv: "d0a25c8721f9da0947be6a8f089815c3"
server_key_version: "encryptionkey@0"
I20260927 18:38:15.378507 3524 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 18:38:15.382807 3539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:15.384797 3524 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:38:15.385015 3524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "825a34c3048c4f8393fe7fbbde109883"
format_stamp: "Formatted at 2026-09-27 18:38:15 on dist-test-slave-9jll"
server_key: "6f3a16f41b7a523f0f1a2cde1953528f"
server_key_iv: "d0a25c8721f9da0947be6a8f089815c3"
server_key_version: "encryptionkey@0"
I20260927 18:38:15.385277 3524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:15.403591 3524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:15.409235 3524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:15.409591 3524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:15.433548 3524 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:45651
I20260927 18:38:15.433579 3600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:45651 every 8 connection(s)
I20260927 18:38:15.435241 3524 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:15.439874 3601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:15.445020 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 3524
I20260927 18:38:15.445286 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:15.446199 2429 external_mini_cluster.cc:1428] Setting key 45103cde31507815253006f4337978a5
I20260927 18:38:15.453760 3601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883: Bootstrap starting.
I20260927 18:38:15.458424 3601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:15.459870 3601 log.cc:824] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:38:15 dist-test-slave-9jll krb5kdc[3511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:15.463907 3601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883: No bootstrap required, opened a new log
I20260927 18:38:15.472669 3601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "825a34c3048c4f8393fe7fbbde109883" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45651 } }
I20260927 18:38:15.473097 3604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:15.448765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.2.95.126:45651)
0927 18:38:15.449819 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:15.449834 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:15.449886 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:38:15.452002 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:15.452035 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:15.452568 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:15.453018 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:15.455065 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:15.456750 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:15.458657 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:15.459551 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:15.459738 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:15.464764 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:15.464775 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:15.464795 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:15.464850 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:15.468216 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:15.469036 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:15.469044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:15.469056 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:15.469261 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:15.469713 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:15.469719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:15.469723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:15.470435 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:15.470786 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:15.471286 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 18:38:15.471834 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":485,"threads_started":1}
I20260927 18:38:15.473124 3601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:15.473282 3601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825a34c3048c4f8393fe7fbbde109883, State: Initialized, Role: FOLLOWER
I20260927 18:38:15.474226 3601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "825a34c3048c4f8393fe7fbbde109883" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45651 } }
I20260927 18:38:15.474537 3601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:15.474684 3601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:15.474865 3601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:15.477610 3601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "825a34c3048c4f8393fe7fbbde109883" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45651 } }
I20260927 18:38:15.478220 3601 leader_election.cc:302] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 825a34c3048c4f8393fe7fbbde109883; no voters:
I20260927 18:38:15.478878 3601 leader_election.cc:288] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:15.478998 3606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:15.479741 3606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [term 1 LEADER]: Becoming Leader. State: Replica: 825a34c3048c4f8393fe7fbbde109883, State: Running, Role: LEADER
I20260927 18:38:15.480444 3606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "825a34c3048c4f8393fe7fbbde109883" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45651 } }
I20260927 18:38:15.481534 3601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:15.483476 3608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "825a34c3048c4f8393fe7fbbde109883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "825a34c3048c4f8393fe7fbbde109883" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45651 } } }
I20260927 18:38:15.483946 3608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:15.486112 3615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:15.486001 3607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 825a34c3048c4f8393fe7fbbde109883. Latest consensus state: current_term: 1 leader_uuid: "825a34c3048c4f8393fe7fbbde109883" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "825a34c3048c4f8393fe7fbbde109883" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45651 } } }
I20260927 18:38:15.486315 3607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:15.492300 3615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:15.503682 3615 catalog_manager.cc:1347] Generated new cluster ID: 7da0b84ba3b141039fab17067c68c1f7
I20260927 18:38:15.503847 3615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:15.526093 3615 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:15.533279 3615 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:15.557060 3615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883: Generated new TSK 0
I20260927 18:38:15.558360 3615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:38:15.613958 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:45651
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:16.009220 3627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:16.009702 3627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:16.009868 3627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:16.018254 3627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:16.018419 3627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:16.018535 3627 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:38:16.018658 3627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:38:16.030689 3627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.95.126:45651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:16.032588 3627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:16.034286 3627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:16.049623 3633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:16.049763 3635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:16.049696 3632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:16.051162 3627 server_base.cc:1034] running on GCE node
I20260927 18:38:16.051860 3627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:16.053017 3627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:16.054337 3627 hybrid_clock.cc:648] HybridClock initialized: now 1790534296054260 us; error 55 us; skew 500 ppm
I20260927 18:38:16.054716 3627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:16 dist-test-slave-9jll krb5kdc[3511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534296, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:16.058622 3627 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:38:16.060535 3627 webserver.cc:460] Webserver started at http://127.2.95.65:38177/ using document root <none> and password file <none>
I20260927 18:38:16.061412 3627 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:16.061540 3627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:16.061964 3627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:16.065527 3627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "a00e31f53d624485b5a62ef18cafa025"
format_stamp: "Formatted at 2026-09-27 18:38:16 on dist-test-slave-9jll"
server_key: "8500b1b1e8bb83922bb11be1d2625302"
server_key_iv: "9b6cb49eacf79b6d52ef05cb3e48b89e"
server_key_version: "encryptionkey@0"
I20260927 18:38:16.066548 3627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "a00e31f53d624485b5a62ef18cafa025"
format_stamp: "Formatted at 2026-09-27 18:38:16 on dist-test-slave-9jll"
server_key: "8500b1b1e8bb83922bb11be1d2625302"
server_key_iv: "9b6cb49eacf79b6d52ef05cb3e48b89e"
server_key_version: "encryptionkey@0"
I20260927 18:38:16.073596 3627 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260927 18:38:16.078306 3642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:16.080112 3627 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:38:16.080324 3627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "a00e31f53d624485b5a62ef18cafa025"
format_stamp: "Formatted at 2026-09-27 18:38:16 on dist-test-slave-9jll"
server_key: "8500b1b1e8bb83922bb11be1d2625302"
server_key_iv: "9b6cb49eacf79b6d52ef05cb3e48b89e"
server_key_version: "encryptionkey@0"
I20260927 18:38:16.080624 3627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:16.118666 3627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:16.124562 3627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:16.124877 3627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:16.128077 3627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:16.128191 3627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:16.128325 3627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:16.128386 3627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:38:16 dist-test-slave-9jll krb5kdc[3511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534296, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:16.155108 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:16.129244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:58369 (local address 127.2.95.126:45651)
0927 18:38:16.130113 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:16.130121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:16.131222 (+ 1101us) server_negotiation.cc:407] Connection header received
0927 18:38:16.134737 (+ 3515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:16.134745 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:16.134861 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:16.135001 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:16.137707 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.138805 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.141381 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.141902 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.142042 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:16.147489 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:16.147496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:16.147500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:16.147545 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:16.150584 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:16.151770 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:16.151775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:16.151778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:16.151927 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:16.152666 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:16.152671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:16.152675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:16.153089 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:16.153235 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:16.154308 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0927 18:38:16.154630 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":555,"threads_started":1}
I20260927 18:38:16.156308 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:16.130043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45651 (local address 127.2.95.65:58369)
0927 18:38:16.130994 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:16.131056 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:16.134254 (+ 3198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:16.135393 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:16.135408 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:16.136357 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:16.137438 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:16.137470 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.139036 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.139041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:16.141207 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:16.141227 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.142181 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.142189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:16.142669 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:16.143642 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:16.143670 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:16.147161 (+ 3491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:16.150788 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:16.150804 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:16.150834 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:16.151623 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:16.152091 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:16.152097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:16.152105 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:16.152525 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:16.153238 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:16.153248 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:16.153791 (+ 543us) client_negotiation.cc:769] Sending connection context
0927 18:38:16.154172 (+ 381us) client_negotiation.cc:240] Negotiation successful
0927 18:38:16.154772 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":369,"threads_started":1}
I20260927 18:38:16.180500 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:16.165160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:56945 (local address 127.2.95.126:45651)
0927 18:38:16.165380 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:16.165388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:16.165999 (+ 611us) server_negotiation.cc:407] Connection header received
0927 18:38:16.166266 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:16.166270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:16.166348 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:16.166470 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:16.167937 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.168764 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.171084 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.171572 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.171686 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:16.174663 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:16.174669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:16.174675 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:16.174727 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:16.177652 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:16.178408 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:16.178412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:16.178414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:16.178548 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:16.178920 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:16.178924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:16.178927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:16.179242 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:16.179380 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:16.179812 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 18:38:16.180052 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:38:16.180603 3730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:16.165141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45651 (local address 127.2.95.65:56945)
0927 18:38:16.165875 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:16.165899 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:16.166101 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:16.166486 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:16.166492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:16.167277 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:16.167737 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:16.167754 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.168906 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.168913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:16.170881 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:16.170902 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.171780 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.171787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:16.171992 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:16.172843 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:16.172870 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:16.174387 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:16.177813 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:16.177822 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:16.177827 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:16.178276 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:16.178667 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:16.178673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:16.178677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:16.178813 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:16.179357 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:16.179361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:16.179551 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 18:38:16.179676 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 18:38:16.180080 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":365,"threads_started":1}
I20260927 18:38:16.189077 3627 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:39593
I20260927 18:38:16.189093 3762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:39593 every 8 connection(s)
I20260927 18:38:16.190822 3627 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:16.193275 3763 heartbeater.cc:347] Connected to a master server at 127.2.95.126:45651
I20260927 18:38:16.193552 3763 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:16.194422 3763 heartbeater.cc:511] Master 127.2.95.126:45651 requested a full tablet report, sending...
I20260927 18:38:16.196884 3556 ts_manager.cc:194] Registered new tserver with Master: a00e31f53d624485b5a62ef18cafa025 (127.2.95.65:39593)
I20260927 18:38:16.198658 3556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:58369
I20260927 18:38:16.201066 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 3627
I20260927 18:38:16.201298 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:16.202127 2429 external_mini_cluster.cc:1428] Setting key af2a9b9bc291a9b8019b31cbf8487928
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:38:16.250319 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:45651
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:16.638757 3767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:16.639149 3767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:16.639259 3767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:16.647230 3767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:16.647377 3767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:16.647472 3767 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:38:16.647560 3767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:38:16.658634 3767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.95.126:45651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:16.660430 3767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:16.662470 3767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:16.678432 3775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:16.678498 3772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:16.678447 3773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:16.679448 3767 server_base.cc:1034] running on GCE node
I20260927 18:38:16.680155 3767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:16.681345 3767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:16.682636 3767 hybrid_clock.cc:648] HybridClock initialized: now 1790534296682554 us; error 52 us; skew 500 ppm
I20260927 18:38:16.683058 3767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:16 dist-test-slave-9jll krb5kdc[3511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534296, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:16.687011 3767 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:38:16.688763 3767 webserver.cc:460] Webserver started at http://127.2.95.66:39659/ using document root <none> and password file <none>
I20260927 18:38:16.689741 3767 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:16.689854 3767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:16.690232 3767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:16.693794 3767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "a6c6499ad14d4283ba03a1b72491927d"
format_stamp: "Formatted at 2026-09-27 18:38:16 on dist-test-slave-9jll"
server_key: "f85b6819c9a52a37f1bb4c3c6e4cc8f0"
server_key_iv: "88a7165e6445fa56b9efc687de1e41dd"
server_key_version: "encryptionkey@0"
I20260927 18:38:16.694659 3767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "a6c6499ad14d4283ba03a1b72491927d"
format_stamp: "Formatted at 2026-09-27 18:38:16 on dist-test-slave-9jll"
server_key: "f85b6819c9a52a37f1bb4c3c6e4cc8f0"
server_key_iv: "88a7165e6445fa56b9efc687de1e41dd"
server_key_version: "encryptionkey@0"
I20260927 18:38:16.700927 3767 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:38:16.705276 3782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:16.707027 3767 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260927 18:38:16.707195 3767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "a6c6499ad14d4283ba03a1b72491927d"
format_stamp: "Formatted at 2026-09-27 18:38:16 on dist-test-slave-9jll"
server_key: "f85b6819c9a52a37f1bb4c3c6e4cc8f0"
server_key_iv: "88a7165e6445fa56b9efc687de1e41dd"
server_key_version: "encryptionkey@0"
I20260927 18:38:16.707432 3767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:16.721705 3767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:16.727195 3767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:16.727523 3767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:16.730670 3767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:16.730765 3767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:16.730902 3767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:16.730962 3767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:38:16 dist-test-slave-9jll krb5kdc[3511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534296, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:16.760476 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:16.731331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:48477 (local address 127.2.95.126:45651)
0927 18:38:16.731908 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:16.731916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:16.733284 (+ 1368us) server_negotiation.cc:407] Connection header received
0927 18:38:16.735596 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:16.735602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:16.735708 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:16.735840 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:16.739331 (+ 3491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.740414 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.743866 (+ 3452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.744490 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.744774 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:16.751066 (+ 6292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:16.751076 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:16.751080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:16.751128 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:16.755382 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:16.756909 (+ 1527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:16.756916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:16.756919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:16.757091 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:16.757836 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:16.757843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:16.757847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:16.758207 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:16.758351 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:16.759706 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0927 18:38:16.760008 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":314,"threads_started":1}
I20260927 18:38:16.762408 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:16.732087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45651 (local address 127.2.95.66:48477)
0927 18:38:16.733065 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:16.733117 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:16.735250 (+ 2133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:16.736248 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:16.736265 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:16.737430 (+ 1165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:16.738937 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:16.738988 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.740686 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.740693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:16.743535 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:16.743564 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.744914 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.744922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:16.745317 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:16.746310 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:16.746359 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:16.750514 (+ 4155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:16.755645 (+ 5131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:16.755665 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:16.755698 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:16.756696 (+ 998us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:16.757220 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:16.757226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:16.757231 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:16.757657 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:16.758365 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:16.758377 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:16.759129 (+ 752us) client_negotiation.cc:769] Sending connection context
0927 18:38:16.759523 (+ 394us) client_negotiation.cc:240] Negotiation successful
0927 18:38:16.760318 (+ 795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":409,"threads_started":1}
I20260927 18:38:16.787425 3767 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:43655
I20260927 18:38:16.787437 3902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:43655 every 8 connection(s)
I20260927 18:38:16.789165 3767 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:16.792982 3903 heartbeater.cc:347] Connected to a master server at 127.2.95.126:45651
I20260927 18:38:16.793287 3903 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:16.793510 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:16.771325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:36475 (local address 127.2.95.126:45651)
0927 18:38:16.771571 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:16.771578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:16.779137 (+ 7559us) server_negotiation.cc:407] Connection header received
0927 18:38:16.779365 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:16.779370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:16.779456 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:16.779582 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:16.781338 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.782469 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.783852 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.784336 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.784473 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:16.787099 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:16.787105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:16.787109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:16.787147 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:16.790169 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:16.791165 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:16.791171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:16.791176 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:16.791344 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:16.791830 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:16.791836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:16.791839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:16.792271 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:16.792407 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:16.792974 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 18:38:16.793222 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 18:38:16.793741 3882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:16.770704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45651 (local address 127.2.95.66:36475)
0927 18:38:16.778932 (+ 8228us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:16.778963 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:16.779227 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:16.779658 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:16.779667 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:16.780454 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:16.781019 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:16.781036 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.782619 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.782625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:16.783691 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:16.783702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:16.784556 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:16.784566 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:16.784786 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:16.785534 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:16.785558 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:16.786853 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:16.790335 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:16.790341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:16.790347 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:16.791023 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:16.791497 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:16.791502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:16.791509 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:16.791686 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:16.792479 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:16.792484 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:16.792679 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 18:38:16.792814 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 18:38:16.793200 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7992,"thread_start_us":7738,"threads_started":1}
I20260927 18:38:16.794384 3903 heartbeater.cc:511] Master 127.2.95.126:45651 requested a full tablet report, sending...
I20260927 18:38:16.794405 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 3767
I20260927 18:38:16.794612 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:16.795239 2429 external_mini_cluster.cc:1428] Setting key d2714233e38f001ddb9166164466e2da
I20260927 18:38:16.796023 3556 ts_manager.cc:194] Registered new tserver with Master: a6c6499ad14d4283ba03a1b72491927d (127.2.95.66:43655)
I20260927 18:38:16.797268 3556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:48477
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:38:16.842370 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:45651
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:38:17.202379 3763 heartbeater.cc:503] Master 127.2.95.126:45651 was elected leader, sending a full tablet report...
W20260927 18:38:17.232362 3907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:17.232761 3907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:17.232898 3907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:17.240978 3907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:17.241163 3907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:17.241300 3907 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:38:17.241423 3907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:38:17.252434 3907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.95.126:45651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:17.254350 3907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:17.255990 3907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:17.272167 3915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:17.272166 3913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:17.272452 3912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:17.273108 3907 server_base.cc:1034] running on GCE node
I20260927 18:38:17.273978 3907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:17.275218 3907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:17.276458 3907 hybrid_clock.cc:648] HybridClock initialized: now 1790534297276365 us; error 65 us; skew 500 ppm
I20260927 18:38:17.276816 3907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:17 dist-test-slave-9jll krb5kdc[3511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534297, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:17.280777 3907 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:38:17.282882 3907 webserver.cc:460] Webserver started at http://127.2.95.67:34217/ using document root <none> and password file <none>
I20260927 18:38:17.283802 3907 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:17.283903 3907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:17.284297 3907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:17.288090 3907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "c5d32c9b812c4e7f982c0c37f0d84977"
format_stamp: "Formatted at 2026-09-27 18:38:17 on dist-test-slave-9jll"
server_key: "153529ae6060c90583c4b3170d12829c"
server_key_iv: "b2dff97181a4ea0b19b308dd6865e04a"
server_key_version: "encryptionkey@0"
I20260927 18:38:17.289002 3907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "c5d32c9b812c4e7f982c0c37f0d84977"
format_stamp: "Formatted at 2026-09-27 18:38:17 on dist-test-slave-9jll"
server_key: "153529ae6060c90583c4b3170d12829c"
server_key_iv: "b2dff97181a4ea0b19b308dd6865e04a"
server_key_version: "encryptionkey@0"
I20260927 18:38:17.295307 3907 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 18:38:17.299387 3922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:17.301059 3907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260927 18:38:17.301250 3907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "c5d32c9b812c4e7f982c0c37f0d84977"
format_stamp: "Formatted at 2026-09-27 18:38:17 on dist-test-slave-9jll"
server_key: "153529ae6060c90583c4b3170d12829c"
server_key_iv: "b2dff97181a4ea0b19b308dd6865e04a"
server_key_version: "encryptionkey@0"
I20260927 18:38:17.301494 3907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:17.318557 3907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:17.323513 3907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:17.323808 3907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:17.327031 3907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:17.327152 3907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:17.327294 3907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:17.327363 3907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:38:17 dist-test-slave-9jll krb5kdc[3511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534297, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:17.358814 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.327794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:34865 (local address 127.2.95.126:45651)
0927 18:38:17.328353 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.328360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.329938 (+ 1578us) server_negotiation.cc:407] Connection header received
0927 18:38:17.332478 (+ 2540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.332486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.332643 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.332783 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:17.335182 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.336458 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.338839 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.339611 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.339773 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.351295 (+ 11522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:17.351304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:17.351310 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:17.351373 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:17.355039 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:17.355937 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:17.355942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:17.355945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:17.356088 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:17.356576 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:17.356580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:17.356582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:17.356888 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:17.357025 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.358108 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.358372 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":317,"threads_started":1}
I20260927 18:38:17.360265 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.328566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45651 (local address 127.2.95.67:34865)
0927 18:38:17.329749 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:17.329802 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:17.332044 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:17.333107 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:17.333125 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:17.334016 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:17.334953 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.334977 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.336665 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.336674 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.338661 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.338683 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.339879 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.339884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.340090 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.346717 (+ 6627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:17.346771 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:17.350825 (+ 4054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:17.355239 (+ 4414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:17.355252 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:17.355273 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:17.355780 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:17.356201 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:17.356207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:17.356211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:17.356440 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:17.357015 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:17.357025 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:17.357489 (+ 464us) client_negotiation.cc:769] Sending connection context
0927 18:38:17.357982 (+ 493us) client_negotiation.cc:240] Negotiation successful
0927 18:38:17.358697 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":561,"threads_started":1}
I20260927 18:38:17.381731 3907 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:33637
I20260927 18:38:17.381767 4042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:33637 every 8 connection(s)
I20260927 18:38:17.383226 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.367108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:36821 (local address 127.2.95.126:45651)
0927 18:38:17.367342 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.367349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.368052 (+ 703us) server_negotiation.cc:407] Connection header received
0927 18:38:17.368199 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.368205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.368309 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.368430 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:17.369618 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.370913 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.372905 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.373630 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.373808 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.375937 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:17.375945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:17.375951 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:17.375998 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:17.379885 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:17.381001 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:17.381005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:17.381008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:17.381184 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:17.381576 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:17.381580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:17.381582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:17.381981 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:17.382121 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.382566 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.382821 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 18:38:17.383368 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.367187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45651 (local address 127.2.95.67:36821)
0927 18:38:17.367936 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:17.367954 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:17.368089 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:17.368490 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:17.368496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:17.369068 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:17.369443 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.369453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.371074 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.371083 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.372742 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.372762 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.373872 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.373880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.374063 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.374853 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:17.374871 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:17.375715 (+ 844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:17.380134 (+ 4419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:17.380138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:17.380141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:17.380847 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:17.381310 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:17.381314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:17.381316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:17.381457 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:17.382105 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:17.382110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:17.382303 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 18:38:17.382456 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 18:38:17.382906 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":367,"threads_started":1}
I20260927 18:38:17.383493 3907 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:38:17.385929 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 3907
I20260927 18:38:17.386119 4043 heartbeater.cc:347] Connected to a master server at 127.2.95.126:45651
I20260927 18:38:17.386173 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:17.386406 4043 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:17.386835 2429 external_mini_cluster.cc:1428] Setting key 3f1f03844a4ae32fa9ee993d2738a8b6
I20260927 18:38:17.387470 4043 heartbeater.cc:511] Master 127.2.95.126:45651 requested a full tablet report, sending...
I20260927 18:38:17.389365 3556 ts_manager.cc:194] Registered new tserver with Master: c5d32c9b812c4e7f982c0c37f0d84977 (127.2.95.67:33637)
I20260927 18:38:17.390429 3556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:34865
I20260927 18:38:17.391184 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:38:17 dist-test-slave-9jll krb5kdc[3511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534297, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:38:17 dist-test-slave-9jll krb5kdc[3511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534297, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:17.444329 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.427700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.2.95.126:45651)
0927 18:38:17.427904 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.427910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.428169 (+ 259us) server_negotiation.cc:407] Connection header received
0927 18:38:17.428370 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.428376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.428461 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.428565 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:17.430245 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.431070 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.432689 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.433174 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.433274 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.438439 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:17.438445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:17.438449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:17.438496 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:17.441345 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:17.442298 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:17.442304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:17.442307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:17.442470 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:17.442878 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:17.442883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:17.442886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:17.443215 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:17.443433 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.443744 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.443940 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:38:17.449512 3556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:38:17.457247 3556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:17.497198 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.472973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.2.95.66:43655)
0927 18:38:17.487198 (+ 14225us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.487207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.488728 (+ 1521us) server_negotiation.cc:407] Connection header received
0927 18:38:17.488847 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.488856 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.489147 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.489365 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:17.490100 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.491387 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.493886 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.495372 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.495537 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.495795 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.496579 (+ 784us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.496815 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14053,"thread_start_us":13937,"threads_started":1}
I20260927 18:38:17.497296 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.472573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:43655 (local address 127.0.0.1:59804)
0927 18:38:17.488590 (+ 16017us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:17.488603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:17.488713 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:17.489367 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:17.489372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:17.489400 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:17.489934 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.489946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.491574 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.491579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.493635 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.493656 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.495986 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.495993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.496268 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.496342 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 18:38:17.496483 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 18:38:17.496789 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15913,"mutex_wait_us":1216,"thread_start_us":13900,"threads_started":1}
I20260927 18:38:17.497359 4058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.472565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:33637 (local address 127.0.0.1:42772)
0927 18:38:17.484479 (+ 11914us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:17.484547 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:17.484895 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:17.487340 (+ 2445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:17.487349 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:17.487386 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:17.488266 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.488284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.490995 (+ 2711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.491000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.493721 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.493735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.495919 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.495924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.496206 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.496373 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 18:38:17.496481 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 18:38:17.496752 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11498,"mutex_wait_us":153,"thread_start_us":15310,"threads_started":1}
I20260927 18:38:17.497385 4056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.473023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39593 (local address 127.0.0.1:50604)
0927 18:38:17.488217 (+ 15194us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:17.488236 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:17.488804 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:17.489577 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:17.489581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:17.489605 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:17.490152 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.490165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.491963 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.491967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.494175 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.494186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.496047 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.496053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.496371 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.496398 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 18:38:17.496471 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 18:38:17.496752 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15032,"mutex_wait_us":169,"thread_start_us":13851,"threads_started":1}
I20260927 18:38:17.498855 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.473313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.2.95.67:33637)
0927 18:38:17.483755 (+ 10442us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.483769 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.485227 (+ 1458us) server_negotiation.cc:407] Connection header received
0927 18:38:17.485385 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.485396 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.485812 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.486074 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:17.488552 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.490512 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.493925 (+ 3413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.495367 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.495494 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.495791 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.498227 (+ 2436us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.498442 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9576,"thread_start_us":9344,"threads_started":1}
I20260927 18:38:17.500648 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.473328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.2.95.65:39593)
0927 18:38:17.486822 (+ 13494us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.486833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.488457 (+ 1624us) server_negotiation.cc:407] Connection header received
0927 18:38:17.488965 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.488971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.489235 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.489439 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:17.490331 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.491760 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.494339 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.495836 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.495995 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.496210 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.500057 (+ 3847us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.500279 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13337,"thread_start_us":13193,"threads_started":1}
I20260927 18:38:17.502633 3833 tablet_service.cc:1467] Processing CreateTablet for tablet 501d8d2c3aca4d439fb79e50f1043e0a (DEFAULT_TABLE table=test-table [id=5bafbbfb10b849528c2ff1512b25c4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:17.502636 3973 tablet_service.cc:1467] Processing CreateTablet for tablet 501d8d2c3aca4d439fb79e50f1043e0a (DEFAULT_TABLE table=test-table [id=5bafbbfb10b849528c2ff1512b25c4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:17.504129 3833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 501d8d2c3aca4d439fb79e50f1043e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:17.504150 3973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 501d8d2c3aca4d439fb79e50f1043e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:17.504544 3693 tablet_service.cc:1467] Processing CreateTablet for tablet 501d8d2c3aca4d439fb79e50f1043e0a (DEFAULT_TABLE table=test-table [id=5bafbbfb10b849528c2ff1512b25c4e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:17.506182 3693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 501d8d2c3aca4d439fb79e50f1043e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:17.517977 4062 tablet_bootstrap.cc:492] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d: Bootstrap starting.
I20260927 18:38:17.521291 4063 tablet_bootstrap.cc:492] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025: Bootstrap starting.
I20260927 18:38:17.523097 4062 tablet_bootstrap.cc:654] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:17.523167 4064 tablet_bootstrap.cc:492] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977: Bootstrap starting.
I20260927 18:38:17.524770 4062 log.cc:824] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:17.525405 4063 tablet_bootstrap.cc:654] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:17.526654 4064 tablet_bootstrap.cc:654] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:17.526633 4063 log.cc:824] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:17.527978 4064 log.cc:824] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:17.528609 4062 tablet_bootstrap.cc:492] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d: No bootstrap required, opened a new log
I20260927 18:38:17.528944 4062 ts_tablet_manager.cc:1397] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 18:38:17.530416 4063 tablet_bootstrap.cc:492] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025: No bootstrap required, opened a new log
I20260927 18:38:17.530732 4063 ts_tablet_manager.cc:1397] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 18:38:17.532462 4064 tablet_bootstrap.cc:492] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977: No bootstrap required, opened a new log
I20260927 18:38:17.532759 4064 ts_tablet_manager.cc:1397] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 18:38:17.540823 4064 raft_consensus.cc:348] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } }
I20260927 18:38:17.540843 4062 raft_consensus.cc:348] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } }
I20260927 18:38:17.541426 4062 raft_consensus.cc:374] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:17.541431 4064 raft_consensus.cc:374] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:17.541532 4064 raft_consensus.cc:729] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d32c9b812c4e7f982c0c37f0d84977, State: Initialized, Role: FOLLOWER
I20260927 18:38:17.541528 4062 raft_consensus.cc:729] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c6499ad14d4283ba03a1b72491927d, State: Initialized, Role: FOLLOWER
I20260927 18:38:17.542388 4064 consensus_queue.cc:260] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } }
I20260927 18:38:17.542394 4062 consensus_queue.cc:260] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } }
I20260927 18:38:17.542168 4063 raft_consensus.cc:348] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } }
I20260927 18:38:17.542958 4063 raft_consensus.cc:374] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:17.543090 4063 raft_consensus.cc:729] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00e31f53d624485b5a62ef18cafa025, State: Initialized, Role: FOLLOWER
I20260927 18:38:17.543792 4043 heartbeater.cc:503] Master 127.2.95.126:45651 was elected leader, sending a full tablet report...
I20260927 18:38:17.544202 4063 consensus_queue.cc:260] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } }
I20260927 18:38:17.544616 4064 ts_tablet_manager.cc:1428] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 18:38:17.545964 3903 heartbeater.cc:503] Master 127.2.95.126:45651 was elected leader, sending a full tablet report...
I20260927 18:38:17.546582 4062 ts_tablet_manager.cc:1428] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260927 18:38:17.546828 4063 ts_tablet_manager.cc:1428] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260927 18:38:17.552522 4069 raft_consensus.cc:482] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:17.552984 4069 raft_consensus.cc:504] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } }
I20260927 18:38:17.555339 4069 leader_election.cc:288] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5d32c9b812c4e7f982c0c37f0d84977 (127.2.95.67:33637), a00e31f53d624485b5a62ef18cafa025 (127.2.95.65:39593)
I20260927 18:38:17.564177 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.556260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:34721 (local address 127.2.95.65:39593)
0927 18:38:17.556426 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.556431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.557344 (+ 913us) server_negotiation.cc:407] Connection header received
0927 18:38:17.557493 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.557497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.557557 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.557642 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:17.558182 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.559222 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.561708 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.562989 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.563182 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.563284 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.563747 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.563899 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 18:38:17.564234 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.555771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:58687 (local address 127.2.95.67:33637)
0927 18:38:17.555997 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.556004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.556692 (+ 688us) server_negotiation.cc:407] Connection header received
0927 18:38:17.556858 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.556864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.556951 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.557048 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:17.557747 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.558868 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.561662 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.562966 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.563116 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.563248 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.563747 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.563899 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:38:17.564378 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.555729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:33637 (local address 127.2.95.66:58687)
0927 18:38:17.556569 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:17.556594 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:17.556728 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:17.557102 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:17.557106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:17.557133 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:17.557554 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.557566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.558996 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.559000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.561420 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.561440 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.563262 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.563271 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.563512 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.563572 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 18:38:17.563691 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 18:38:17.563911 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":95,"thread_start_us":1183,"threads_started":1}
I20260927 18:38:17.564590 4072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.555764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39593 (local address 127.2.95.66:34721)
0927 18:38:17.557218 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:17.557240 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:17.557366 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:17.557690 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:17.557695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:17.557720 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:17.558014 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.558022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.559335 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.559340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.561518 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.561530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.563304 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.563311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.563524 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.563570 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 18:38:17.563679 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 18:38:17.563911 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1337,"mutex_wait_us":55,"thread_start_us":489,"threads_started":1}
I20260927 18:38:17.565889 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "501d8d2c3aca4d439fb79e50f1043e0a" candidate_uuid: "a6c6499ad14d4283ba03a1b72491927d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a00e31f53d624485b5a62ef18cafa025" is_pre_election: true
I20260927 18:38:17.565889 3993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "501d8d2c3aca4d439fb79e50f1043e0a" candidate_uuid: "a6c6499ad14d4283ba03a1b72491927d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" is_pre_election: true
I20260927 18:38:17.566478 3713 raft_consensus.cc:2454] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6c6499ad14d4283ba03a1b72491927d in term 0.
I20260927 18:38:17.566478 3993 raft_consensus.cc:2454] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6c6499ad14d4283ba03a1b72491927d in term 0.
I20260927 18:38:17.567202 3783 leader_election.cc:302] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a00e31f53d624485b5a62ef18cafa025, a6c6499ad14d4283ba03a1b72491927d; no voters:
I20260927 18:38:17.567692 4069 raft_consensus.cc:2781] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:17.567832 4069 raft_consensus.cc:482] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:17.567931 4069 raft_consensus.cc:3037] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:17.570613 4069 raft_consensus.cc:504] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } }
I20260927 18:38:17.571395 4069 leader_election.cc:288] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [CANDIDATE]: Term 1 election: Requested vote from peers c5d32c9b812c4e7f982c0c37f0d84977 (127.2.95.67:33637), a00e31f53d624485b5a62ef18cafa025 (127.2.95.65:39593)
I20260927 18:38:17.571916 3993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "501d8d2c3aca4d439fb79e50f1043e0a" candidate_uuid: "a6c6499ad14d4283ba03a1b72491927d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d32c9b812c4e7f982c0c37f0d84977"
I20260927 18:38:17.571921 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "501d8d2c3aca4d439fb79e50f1043e0a" candidate_uuid: "a6c6499ad14d4283ba03a1b72491927d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a00e31f53d624485b5a62ef18cafa025"
I20260927 18:38:17.572155 3993 raft_consensus.cc:3037] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:17.572219 3713 raft_consensus.cc:3037] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:17.575693 3993 raft_consensus.cc:2454] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6c6499ad14d4283ba03a1b72491927d in term 1.
I20260927 18:38:17.575690 3713 raft_consensus.cc:2454] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6c6499ad14d4283ba03a1b72491927d in term 1.
I20260927 18:38:17.576238 3784 leader_election.cc:302] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a00e31f53d624485b5a62ef18cafa025, a6c6499ad14d4283ba03a1b72491927d, c5d32c9b812c4e7f982c0c37f0d84977; no voters:
I20260927 18:38:17.576622 4069 raft_consensus.cc:2781] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:17.577142 4069 raft_consensus.cc:686] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 1 LEADER]: Becoming Leader. State: Replica: a6c6499ad14d4283ba03a1b72491927d, State: Running, Role: LEADER
I20260927 18:38:17.577816 4069 consensus_queue.cc:237] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } }
I20260927 18:38:17.583310 3556 catalog_manager.cc:5495] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d reported cstate change: term changed from 0 to 1, leader changed from <none> to a6c6499ad14d4283ba03a1b72491927d (127.2.95.66). New cstate: current_term: 1 leader_uuid: "a6c6499ad14d4283ba03a1b72491927d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } health_report { overall_health: UNKNOWN } } }
I20260927 18:38:17.613031 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.606490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.2.95.66:43655)
0927 18:38:17.606675 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.606681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.606816 (+ 135us) server_negotiation.cc:407] Connection header received
0927 18:38:17.607138 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.607143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.607236 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.607349 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:17.607735 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.608564 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.610196 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.610731 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.610849 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.611435 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:17.612214 (+ 779us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:17.612335 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.612524 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.612619 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:38:17.623082 3993 raft_consensus.cc:1261] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977 [term 1 FOLLOWER]: Refusing update from remote peer a6c6499ad14d4283ba03a1b72491927d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:17.623076 3713 raft_consensus.cc:1261] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025 [term 1 FOLLOWER]: Refusing update from remote peer a6c6499ad14d4283ba03a1b72491927d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:17.624200 4073 consensus_queue.cc:1035] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:17.624627 4069 consensus_queue.cc:1035] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:38:17.635530 4044 tablet.cc:2369] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:38:17.640025 4078 mvcc.cc:205] Tried to move back new op lower bound from 7334028483052412928 to 7334028482883330048. Current Snapshot: MvccSnapshot[applied={T|T < 7334028483052412928}]
I20260927 18:38:17.660866 3555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:17.661238 3555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:17.666991 3555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883: Sending DeleteTablet for 3 replicas of tablet 501d8d2c3aca4d439fb79e50f1043e0a
I20260927 18:38:17.668776 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet 501d8d2c3aca4d439fb79e50f1043e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:17 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:59804
I20260927 18:38:17.668962 3973 tablet_service.cc:1514] Processing DeleteTablet for tablet 501d8d2c3aca4d439fb79e50f1043e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:17 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:42772
I20260927 18:38:17.671725 4086 tablet_replica.cc:330] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d: stopping tablet replica
I20260927 18:38:17.672115 3693 tablet_service.cc:1514] Processing DeleteTablet for tablet 501d8d2c3aca4d439fb79e50f1043e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:17 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:50604
I20260927 18:38:17.672523 4087 tablet_replica.cc:330] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977: stopping tablet replica
I20260927 18:38:17.673136 4087 raft_consensus.cc:2229] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:17.673883 4087 raft_consensus.cc:2258] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:17.674463 3555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:38:17.675444 3555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:17.675920 4086 raft_consensus.cc:2229] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:38:17.676857 4086 raft_consensus.cc:2258] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:17.677893 4088 tablet_replica.cc:330] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025: stopping tablet replica
I20260927 18:38:17.678448 4088 raft_consensus.cc:2229] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:17.679075 4088 raft_consensus.cc:2258] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:17.679920 4086 ts_tablet_manager.cc:1905] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:17.681970 4088 ts_tablet_manager.cc:1905] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:17.685879 4087 ts_tablet_manager.cc:1905] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:17.687188 3833 tablet_service.cc:1467] Processing CreateTablet for tablet 8250e1865dea45bfa18c6d5110a5bd9b (DEFAULT_TABLE table=test-table [id=b2ab55f256d84df99ea0d5a4afa8f856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:17.687742 3833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8250e1865dea45bfa18c6d5110a5bd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:17.690395 4088 ts_tablet_manager.cc:1918] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:17.690694 4088 log.cc:1196] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/501d8d2c3aca4d439fb79e50f1043e0a
I20260927 18:38:17.691370 4088 ts_tablet_manager.cc:1939] T 501d8d2c3aca4d439fb79e50f1043e0a P a00e31f53d624485b5a62ef18cafa025: Deleting consensus metadata
I20260927 18:38:17.693992 3540 catalog_manager.cc:4841] TS a00e31f53d624485b5a62ef18cafa025 (127.2.95.65:39593): tablet 501d8d2c3aca4d439fb79e50f1043e0a (table test-table [id=5bafbbfb10b849528c2ff1512b25c4e4]) successfully deleted
I20260927 18:38:17.694334 3973 tablet_service.cc:1467] Processing CreateTablet for tablet 8250e1865dea45bfa18c6d5110a5bd9b (DEFAULT_TABLE table=test-table [id=b2ab55f256d84df99ea0d5a4afa8f856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:17.694854 3973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8250e1865dea45bfa18c6d5110a5bd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:17.697254 4062 tablet_bootstrap.cc:492] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d: Bootstrap starting.
I20260927 18:38:17.697911 4087 ts_tablet_manager.cc:1918] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:17.698127 4087 log.cc:1196] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/501d8d2c3aca4d439fb79e50f1043e0a
I20260927 18:38:17.698757 4087 ts_tablet_manager.cc:1939] T 501d8d2c3aca4d439fb79e50f1043e0a P c5d32c9b812c4e7f982c0c37f0d84977: Deleting consensus metadata
I20260927 18:38:17.701124 4086 ts_tablet_manager.cc:1918] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:17.701254 3541 catalog_manager.cc:4841] TS c5d32c9b812c4e7f982c0c37f0d84977 (127.2.95.67:33637): tablet 501d8d2c3aca4d439fb79e50f1043e0a (table test-table [id=5bafbbfb10b849528c2ff1512b25c4e4]) successfully deleted
I20260927 18:38:17.701354 4086 log.cc:1196] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/501d8d2c3aca4d439fb79e50f1043e0a
I20260927 18:38:17.701489 3693 tablet_service.cc:1467] Processing CreateTablet for tablet 8250e1865dea45bfa18c6d5110a5bd9b (DEFAULT_TABLE table=test-table [id=b2ab55f256d84df99ea0d5a4afa8f856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:17.702031 4086 ts_tablet_manager.cc:1939] T 501d8d2c3aca4d439fb79e50f1043e0a P a6c6499ad14d4283ba03a1b72491927d: Deleting consensus metadata
I20260927 18:38:17.702220 3693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8250e1865dea45bfa18c6d5110a5bd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:17.703502 4062 tablet_bootstrap.cc:654] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:17.704232 3542 catalog_manager.cc:4841] TS a6c6499ad14d4283ba03a1b72491927d (127.2.95.66:43655): tablet 501d8d2c3aca4d439fb79e50f1043e0a (table test-table [id=5bafbbfb10b849528c2ff1512b25c4e4]) successfully deleted
I20260927 18:38:17.707672 4064 tablet_bootstrap.cc:492] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977: Bootstrap starting.
I20260927 18:38:17.710271 4064 tablet_bootstrap.cc:654] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:17.712262 4063 tablet_bootstrap.cc:492] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025: Bootstrap starting.
I20260927 18:38:17.712605 4062 tablet_bootstrap.cc:492] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d: No bootstrap required, opened a new log
I20260927 18:38:17.712905 4062 ts_tablet_manager.cc:1397] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20260927 18:38:17.714102 4063 tablet_bootstrap.cc:654] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:17.715917 4062 raft_consensus.cc:348] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:17.716420 4062 raft_consensus.cc:374] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:17.716591 4062 raft_consensus.cc:729] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c6499ad14d4283ba03a1b72491927d, State: Initialized, Role: FOLLOWER
I20260927 18:38:17.717010 4062 consensus_queue.cc:260] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:17.717833 4062 ts_tablet_manager.cc:1428] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260927 18:38:17.724956 4063 tablet_bootstrap.cc:492] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025: No bootstrap required, opened a new log
I20260927 18:38:17.725178 4063 ts_tablet_manager.cc:1397] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260927 18:38:17.725595 4064 tablet_bootstrap.cc:492] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977: No bootstrap required, opened a new log
I20260927 18:38:17.725785 4064 ts_tablet_manager.cc:1397] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260927 18:38:17.726221 4063 raft_consensus.cc:348] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:17.726486 4063 raft_consensus.cc:374] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:17.726583 4063 raft_consensus.cc:729] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00e31f53d624485b5a62ef18cafa025, State: Initialized, Role: FOLLOWER
I20260927 18:38:17.726495 4064 raft_consensus.cc:348] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:17.726723 4064 raft_consensus.cc:374] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:17.726821 4064 raft_consensus.cc:729] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d32c9b812c4e7f982c0c37f0d84977, State: Initialized, Role: FOLLOWER
I20260927 18:38:17.726907 4063 consensus_queue.cc:260] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:17.727080 4064 consensus_queue.cc:260] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:17.727597 4063 ts_tablet_manager.cc:1428] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:38:17.727635 4064 ts_tablet_manager.cc:1428] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20260927 18:38:17.793195 3904 tablet.cc:2369] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:38:17.886054 4070 raft_consensus.cc:482] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:17.886463 4070 raft_consensus.cc:504] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:17.888974 4070 leader_election.cc:288] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6c6499ad14d4283ba03a1b72491927d (127.2.95.66:43655), c5d32c9b812c4e7f982c0c37f0d84977 (127.2.95.67:33637)
I20260927 18:38:17.897338 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.890184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:56679 (local address 127.2.95.67:33637)
0927 18:38:17.890384 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.890390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.891104 (+ 714us) server_negotiation.cc:407] Connection header received
0927 18:38:17.891284 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.891290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.891359 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.891435 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:17.892086 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.893219 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.895538 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.896353 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.896478 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.896591 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.896912 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.897055 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 18:38:17.897408 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.889634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:33637 (local address 127.2.95.65:56679)
0927 18:38:17.890992 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:17.891009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:17.891154 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:17.891463 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:17.891467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:17.891495 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:17.891894 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.891909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.893363 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.893368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.895418 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.895429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.896579 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.896593 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.896754 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.896783 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:38:17.896870 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:38:17.897021 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1238,"thread_start_us":1044,"threads_started":1}
I20260927 18:38:17.898257 3993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8250e1865dea45bfa18c6d5110a5bd9b" candidate_uuid: "a00e31f53d624485b5a62ef18cafa025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" is_pre_election: true
I20260927 18:38:17.898545 3993 raft_consensus.cc:2454] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a00e31f53d624485b5a62ef18cafa025 in term 0.
I20260927 18:38:17.898612 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.889409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:43655 (local address 127.2.95.65:57141)
0927 18:38:17.890273 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:17.890294 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:17.890502 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:17.891148 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:17.891153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:17.891180 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:17.891653 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.891667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.892924 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.892930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.895285 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.895302 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.897295 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.897301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.897924 (+ 623us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.897963 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 18:38:17.898060 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:38:17.898209 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":54,"thread_start_us":543,"threads_started":1}
I20260927 18:38:17.899144 3644 leader_election.cc:302] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a00e31f53d624485b5a62ef18cafa025, c5d32c9b812c4e7f982c0c37f0d84977; no voters:
I20260927 18:38:17.899518 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.889418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:57141 (local address 127.2.95.66:43655)
0927 18:38:17.889710 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.889723 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.890414 (+ 691us) server_negotiation.cc:407] Connection header received
0927 18:38:17.890715 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.890722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.890835 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.890908 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:17.891844 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.892784 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.895483 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.896894 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.897043 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.897137 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:17.899051 (+ 1914us) server_negotiation.cc:299] Negotiation successful
0927 18:38:17.899212 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 18:38:17.899578 4070 raft_consensus.cc:2781] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:17.899703 4070 raft_consensus.cc:482] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:17.899783 4070 raft_consensus.cc:3037] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:17.900501 3853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8250e1865dea45bfa18c6d5110a5bd9b" candidate_uuid: "a00e31f53d624485b5a62ef18cafa025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c6499ad14d4283ba03a1b72491927d" is_pre_election: true
I20260927 18:38:17.901103 3853 raft_consensus.cc:2454] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a00e31f53d624485b5a62ef18cafa025 in term 0.
I20260927 18:38:17.902992 4070 raft_consensus.cc:504] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:17.903776 4070 leader_election.cc:288] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [CANDIDATE]: Term 1 election: Requested vote from peers a6c6499ad14d4283ba03a1b72491927d (127.2.95.66:43655), c5d32c9b812c4e7f982c0c37f0d84977 (127.2.95.67:33637)
I20260927 18:38:17.904294 3853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8250e1865dea45bfa18c6d5110a5bd9b" candidate_uuid: "a00e31f53d624485b5a62ef18cafa025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c6499ad14d4283ba03a1b72491927d"
I20260927 18:38:17.904440 3993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8250e1865dea45bfa18c6d5110a5bd9b" candidate_uuid: "a00e31f53d624485b5a62ef18cafa025" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d32c9b812c4e7f982c0c37f0d84977"
I20260927 18:38:17.904546 3853 raft_consensus.cc:3037] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:17.904688 3993 raft_consensus.cc:3037] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:17.908016 3993 raft_consensus.cc:2454] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a00e31f53d624485b5a62ef18cafa025 in term 1.
I20260927 18:38:17.908123 3853 raft_consensus.cc:2454] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a00e31f53d624485b5a62ef18cafa025 in term 1.
I20260927 18:38:17.908629 3644 leader_election.cc:302] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a00e31f53d624485b5a62ef18cafa025, a6c6499ad14d4283ba03a1b72491927d, c5d32c9b812c4e7f982c0c37f0d84977; no voters:
I20260927 18:38:17.909050 4070 raft_consensus.cc:2781] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:17.909777 4070 raft_consensus.cc:686] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 1 LEADER]: Becoming Leader. State: Replica: a00e31f53d624485b5a62ef18cafa025, State: Running, Role: LEADER
I20260927 18:38:17.910259 4070 consensus_queue.cc:237] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:17.915066 3555 catalog_manager.cc:5495] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 reported cstate change: term changed from 0 to 1, leader changed from <none> to a00e31f53d624485b5a62ef18cafa025 (127.2.95.65). New cstate: current_term: 1 leader_uuid: "a00e31f53d624485b5a62ef18cafa025" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } health_report { overall_health: UNKNOWN } } }
W20260927 18:38:17.944031 3764 tablet.cc:2369] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 18:38:17 dist-test-slave-9jll krb5kdc[3511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:18.007997 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.989526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.2.95.126:45651)
0927 18:38:17.991130 (+ 1604us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:17.991135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:17.991158 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:38:17.991209 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:17.991212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:17.991272 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:17.991373 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:17.993201 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.994169 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.996021 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.996468 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.996581 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.001384 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:18.001390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:18.001393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:18.001431 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:18.004434 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:18.005622 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:18.005627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:18.005630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:18.005854 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:18.006250 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:18.006256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:18.006260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:18.006640 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:18.006804 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.007291 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.007548 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1501,"thread_start_us":1360,"threads_started":1}
I20260927 18:38:18.008059 4101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:17.989508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45651 (local address 127.0.0.1:34514)
0927 18:38:17.990718 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:17.990743 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:17.990956 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:17.991450 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:17.991456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:17.992614 (+ 1158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:17.993000 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.993014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.994318 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.994323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.995850 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:17.995865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:17.996715 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:17.996727 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:17.996907 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:17.997862 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:17.997890 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:18.001086 (+ 3196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:18.004635 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:18.004657 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:18.004670 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:18.005484 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:18.005972 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:18.005978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:18.005982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:18.006144 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:18.006788 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:18.006799 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:18.007019 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 18:38:18.007190 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 18:38:18.007696 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"mutex_wait_us":170,"spinlock_wait_cycles":9600,"thread_start_us":798,"threads_started":1}
I20260927 18:38:18.014899 3555 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:34514:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:38:18.016170 3555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:18.024407 3693 tablet_service.cc:1467] Processing CreateTablet for tablet 9804fe8915ab4e798e04037119e93180 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e81cba14d99f4327bc66a59aa747b13e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:38:18.024528 3833 tablet_service.cc:1467] Processing CreateTablet for tablet 9804fe8915ab4e798e04037119e93180 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e81cba14d99f4327bc66a59aa747b13e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:38:18.025206 3833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9804fe8915ab4e798e04037119e93180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:18.025213 3693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9804fe8915ab4e798e04037119e93180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:18.032429 4062 tablet_bootstrap.cc:492] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d: Bootstrap starting.
I20260927 18:38:18.034058 3973 tablet_service.cc:1467] Processing CreateTablet for tablet 9804fe8915ab4e798e04037119e93180 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e81cba14d99f4327bc66a59aa747b13e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:38:18.034674 4062 tablet_bootstrap.cc:654] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:18.034725 4063 tablet_bootstrap.cc:492] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025: Bootstrap starting.
I20260927 18:38:18.034759 3973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9804fe8915ab4e798e04037119e93180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:18.036631 4063 tablet_bootstrap.cc:654] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:18.042769 4064 tablet_bootstrap.cc:492] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: Bootstrap starting.
I20260927 18:38:18.044729 4064 tablet_bootstrap.cc:654] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:18.044903 4063 tablet_bootstrap.cc:492] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025: No bootstrap required, opened a new log
I20260927 18:38:18.045102 4063 ts_tablet_manager.cc:1397] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260927 18:38:18.046041 4062 tablet_bootstrap.cc:492] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d: No bootstrap required, opened a new log
I20260927 18:38:18.046036 4063 raft_consensus.cc:348] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:18.046311 4063 raft_consensus.cc:374] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:18.046401 4063 raft_consensus.cc:729] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00e31f53d624485b5a62ef18cafa025, State: Initialized, Role: FOLLOWER
I20260927 18:38:18.046409 4062 ts_tablet_manager.cc:1397] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20260927 18:38:18.046681 4063 consensus_queue.cc:260] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:18.047307 4063 ts_tablet_manager.cc:1428] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:38:18.048663 4070 tablet_replica.cc:439] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } }
I20260927 18:38:18.049017 4070 tablet_replica.cc:442] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:18.048727 4062 raft_consensus.cc:348] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:18.049206 4062 raft_consensus.cc:374] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:18.049342 4062 raft_consensus.cc:729] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c6499ad14d4283ba03a1b72491927d, State: Initialized, Role: FOLLOWER
I20260927 18:38:18.049736 4062 consensus_queue.cc:260] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:18.050555 4062 ts_tablet_manager.cc:1428] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260927 18:38:18.051029 4073 tablet_replica.cc:439] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } }
I20260927 18:38:18.051397 4073 tablet_replica.cc:442] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:18.054863 4064 tablet_bootstrap.cc:492] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: No bootstrap required, opened a new log
I20260927 18:38:18.055106 4064 ts_tablet_manager.cc:1397] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 18:38:18.056017 4064 raft_consensus.cc:348] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:18.056331 4064 raft_consensus.cc:374] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:18.056429 4064 raft_consensus.cc:729] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d32c9b812c4e7f982c0c37f0d84977, State: Initialized, Role: FOLLOWER
I20260927 18:38:18.056721 4064 consensus_queue.cc:260] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:18.057302 4068 tablet_replica.cc:439] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } }
I20260927 18:38:18.057627 4068 tablet_replica.cc:442] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:18.058040 4064 ts_tablet_manager.cc:1428] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:38:18.274588 4068 raft_consensus.cc:482] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:18.274971 4068 raft_consensus.cc:504] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:18.276958 4068 leader_election.cc:288] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a00e31f53d624485b5a62ef18cafa025 (127.2.95.65:39593), a6c6499ad14d4283ba03a1b72491927d (127.2.95.66:43655)
I20260927 18:38:18.283943 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.277394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:45459 (local address 127.2.95.66:43655)
0927 18:38:18.277622 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.277630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.278542 (+ 912us) server_negotiation.cc:407] Connection header received
0927 18:38:18.278714 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.278720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.278805 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.278909 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:18.279477 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.280278 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.282071 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.282938 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.283040 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.283147 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.283490 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.283644 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 18:38:18.284060 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.277243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:43655 (local address 127.2.95.67:45459)
0927 18:38:18.278405 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:18.278438 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:18.278563 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:18.278941 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:18.278946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:18.278976 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:18.279322 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:18.279332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.280411 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.280415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:18.281902 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:18.281916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.283110 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.283117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:18.283329 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.283363 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:38:18.283446 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 18:38:18.283587 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":92,"thread_start_us":827,"threads_started":1}
I20260927 18:38:18.284109 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.277243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39593 (local address 127.2.95.67:54745)
0927 18:38:18.277863 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:18.277890 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:18.278085 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:18.278565 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:18.278571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:18.278615 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:18.278916 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:18.278929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.280145 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.280151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:18.281834 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:18.281847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.283257 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.283262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:18.283411 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.283442 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:38:18.283530 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:38:18.283674 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":1,"thread_start_us":342,"threads_started":1}
I20260927 18:38:18.284511 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.277394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:54745 (local address 127.2.95.65:39593)
0927 18:38:18.278007 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.278012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.278030 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:38:18.278270 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.278276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.278365 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.278517 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:18.279098 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.280014 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.282002 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.283057 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.283190 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.283306 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.283572 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.283715 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":392,"threads_started":1}
I20260927 18:38:18.284783 3853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9804fe8915ab4e798e04037119e93180" candidate_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c6499ad14d4283ba03a1b72491927d" is_pre_election: true
I20260927 18:38:18.285075 3853 raft_consensus.cc:2454] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d32c9b812c4e7f982c0c37f0d84977 in term 0.
I20260927 18:38:18.285084 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9804fe8915ab4e798e04037119e93180" candidate_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a00e31f53d624485b5a62ef18cafa025" is_pre_election: true
I20260927 18:38:18.285317 3713 raft_consensus.cc:2454] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5d32c9b812c4e7f982c0c37f0d84977 in term 0.
I20260927 18:38:18.285728 3925 leader_election.cc:302] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6c6499ad14d4283ba03a1b72491927d, c5d32c9b812c4e7f982c0c37f0d84977; no voters:
I20260927 18:38:18.286183 4068 raft_consensus.cc:2781] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:18.286303 4068 raft_consensus.cc:482] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:18.286412 4068 raft_consensus.cc:3037] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:18.288877 4068 raft_consensus.cc:504] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:18.289523 4068 leader_election.cc:288] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [CANDIDATE]: Term 1 election: Requested vote from peers a00e31f53d624485b5a62ef18cafa025 (127.2.95.65:39593), a6c6499ad14d4283ba03a1b72491927d (127.2.95.66:43655)
I20260927 18:38:18.290063 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9804fe8915ab4e798e04037119e93180" candidate_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a00e31f53d624485b5a62ef18cafa025"
I20260927 18:38:18.290119 3853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9804fe8915ab4e798e04037119e93180" candidate_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6c6499ad14d4283ba03a1b72491927d"
I20260927 18:38:18.290354 3853 raft_consensus.cc:3037] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:18.291195 3713 raft_consensus.cc:3037] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:18.293723 3853 raft_consensus.cc:2454] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d32c9b812c4e7f982c0c37f0d84977 in term 1.
I20260927 18:38:18.294149 3713 raft_consensus.cc:2454] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5d32c9b812c4e7f982c0c37f0d84977 in term 1.
I20260927 18:38:18.294219 3925 leader_election.cc:302] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6c6499ad14d4283ba03a1b72491927d, c5d32c9b812c4e7f982c0c37f0d84977; no voters:
I20260927 18:38:18.294587 4068 raft_consensus.cc:2781] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:18.295017 4068 raft_consensus.cc:686] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [term 1 LEADER]: Becoming Leader. State: Replica: c5d32c9b812c4e7f982c0c37f0d84977, State: Running, Role: LEADER
I20260927 18:38:18.295491 4068 consensus_queue.cc:237] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } }
I20260927 18:38:18.297897 4109 tablet_replica.cc:439] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: TxnStatusTablet state changed. Reason: New leader c5d32c9b812c4e7f982c0c37f0d84977. Latest consensus state: current_term: 1 leader_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } }
I20260927 18:38:18.298373 4109 tablet_replica.cc:442] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:38:18.299008 4111 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:38:18.299213 4111 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:38:18.299522 3556 catalog_manager.cc:5495] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5d32c9b812c4e7f982c0c37f0d84977 (127.2.95.67). New cstate: current_term: 1 leader_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } health_report { overall_health: HEALTHY } } }
I20260927 18:38:18.344633 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.338826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.2.95.67:33637)
0927 18:38:18.339040 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.339045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.339061 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:38:18.339156 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.339160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.339251 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.339385 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:18.339794 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.340606 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.342064 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.342577 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.342699 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.343207 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:18.343907 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:18.344064 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.344162 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.344253 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:38:18.344607 4101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.338344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:33637 (local address 127.0.0.1:42784)
0927 18:38:18.338616 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:18.338642 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:18.338805 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:18.339428 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:18.339434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:18.339450 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:38:18.339663 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:18.339672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.340756 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.340762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:18.341922 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:18.341935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.342739 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.342744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:18.342929 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.343009 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:38:18.344035 (+ 1026us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:38:18.344060 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:38:18.344139 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:38:18.344312 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 18:38:18.348632 4070 consensus_queue.cc:1035] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:18.351527 3852 raft_consensus.cc:1261] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d [term 1 FOLLOWER]: Refusing update from remote peer c5d32c9b812c4e7f982c0c37f0d84977: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:18.352416 4109 consensus_queue.cc:1035] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:18.352943 3713 raft_consensus.cc:1261] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025 [term 1 FOLLOWER]: Refusing update from remote peer c5d32c9b812c4e7f982c0c37f0d84977: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:18.354966 4109 consensus_queue.cc:1035] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:18.367240 4073 tablet_replica.cc:439] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d: TxnStatusTablet state changed. Reason: New leader c5d32c9b812c4e7f982c0c37f0d84977. Latest consensus state: current_term: 1 leader_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } }
I20260927 18:38:18.367540 4073 tablet_replica.cc:442] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:18.372176 4073 tablet_replica.cc:439] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } }
I20260927 18:38:18.372493 4073 tablet_replica.cc:442] T 9804fe8915ab4e798e04037119e93180 P a6c6499ad14d4283ba03a1b72491927d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:18.375962 4068 tablet_replica.cc:439] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } }
I20260927 18:38:18.376425 4068 tablet_replica.cc:442] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:38:18.384496 4126 consensus_queue.cc:1035] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 18:38:18.387840 4068 tablet_replica.cc:439] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } }
I20260927 18:38:18.388154 4068 tablet_replica.cc:442] T 9804fe8915ab4e798e04037119e93180 P c5d32c9b812c4e7f982c0c37f0d84977: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:38:18.388006 4094 tablet_replica.cc:439] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025: TxnStatusTablet state changed. Reason: New leader c5d32c9b812c4e7f982c0c37f0d84977. Latest consensus state: current_term: 1 leader_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } }
I20260927 18:38:18.388386 4094 tablet_replica.cc:442] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:18.415619 4070 tablet_replica.cc:439] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00e31f53d624485b5a62ef18cafa025" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39593 } } peers { permanent_uuid: "a6c6499ad14d4283ba03a1b72491927d" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43655 } } peers { permanent_uuid: "c5d32c9b812c4e7f982c0c37f0d84977" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33637 } } }
I20260927 18:38:18.415947 4070 tablet_replica.cc:442] T 9804fe8915ab4e798e04037119e93180 P a00e31f53d624485b5a62ef18cafa025: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:18.428377 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.403275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.2.95.65:39593)
0927 18:38:18.403476 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.403483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.412510 (+ 9027us) server_negotiation.cc:407] Connection header received
0927 18:38:18.416812 (+ 4302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.416819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.416941 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.417082 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:18.418076 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.419244 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.424916 (+ 5672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.425653 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.425835 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.426604 (+ 769us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:18.427592 (+ 988us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:18.427753 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.427953 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.428065 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:38:18.450969 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.442550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:37313 (local address 127.2.95.67:33637)
0927 18:38:18.442750 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.442757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.443422 (+ 665us) server_negotiation.cc:407] Connection header received
0927 18:38:18.443642 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.443649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.443751 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.443847 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:18.444405 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.445603 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.447965 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.448744 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.448879 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.449448 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:18.450194 (+ 746us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:18.450284 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.450492 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.450611 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:38:18.451917 4133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.442537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:33637 (local address 127.2.95.65:37313)
0927 18:38:18.443280 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:18.443305 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:18.443486 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:18.443874 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:18.443879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:18.443892 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:38:18.444219 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:18.444231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.445821 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.445828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:18.447772 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:18.447794 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.448929 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.448938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:18.449146 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.449240 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:38:18.450336 (+ 1096us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:38:18.450349 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 18:38:18.451297 (+ 948us) client_negotiation.cc:240] Negotiation successful
0927 18:38:18.451475 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":358,"threads_started":1}
I20260927 18:38:18.512347 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.503942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:60437 (local address 127.2.95.65:39593)
0927 18:38:18.504171 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.504179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.504835 (+ 656us) server_negotiation.cc:407] Connection header received
0927 18:38:18.505063 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.505069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.505193 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.505290 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:18.506361 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.507468 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.510012 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.510577 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.510714 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.511430 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:18.511608 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:18.511676 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.511941 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.512028 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:38:18.512775 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.503891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39593 (local address 127.2.95.67:60437)
0927 18:38:18.504728 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:18.504745 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:18.504904 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:18.505475 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:18.505480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:18.505495 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:38:18.506094 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:18.506111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.507667 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.507674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:18.509851 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:18.509869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.510784 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.510791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:18.511067 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.511208 (+ 141us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:38:18.511777 (+ 569us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:38:18.511790 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 18:38:18.511908 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 18:38:18.512155 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":435,"threads_started":1}
I20260927 18:38:18.566907 3555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:18.567257 3555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:18.571656 3555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 825a34c3048c4f8393fe7fbbde109883: Sending DeleteTablet for 3 replicas of tablet 8250e1865dea45bfa18c6d5110a5bd9b
I20260927 18:38:18.572743 3693 tablet_service.cc:1514] Processing DeleteTablet for tablet 8250e1865dea45bfa18c6d5110a5bd9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:18 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:50604
I20260927 18:38:18.572765 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet 8250e1865dea45bfa18c6d5110a5bd9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:18 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:59804
I20260927 18:38:18.574671 3973 tablet_service.cc:1514] Processing DeleteTablet for tablet 8250e1865dea45bfa18c6d5110a5bd9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:18 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:42772
I20260927 18:38:18.582324 4142 tablet_replica.cc:330] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977: stopping tablet replica
I20260927 18:38:18.582636 4142 raft_consensus.cc:2229] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:18.583002 4142 raft_consensus.cc:2258] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:18.584059 4140 tablet_replica.cc:330] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025: stopping tablet replica
I20260927 18:38:18.585057 4140 raft_consensus.cc:2229] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:38:18.586400 4140 raft_consensus.cc:2258] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:18.587440 4141 tablet_replica.cc:330] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d: stopping tablet replica
I20260927 18:38:18.587755 4141 raft_consensus.cc:2229] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:18.588032 4141 raft_consensus.cc:2258] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:18.589593 4140 ts_tablet_manager.cc:1905] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:18.591992 4141 ts_tablet_manager.cc:1905] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:18.594810 4142 ts_tablet_manager.cc:1905] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:18.601751 4140 ts_tablet_manager.cc:1918] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:38:18.601998 4142 ts_tablet_manager.cc:1918] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:38:18.602149 4142 log.cc:1196] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/8250e1865dea45bfa18c6d5110a5bd9b
I20260927 18:38:18.602669 4140 log.cc:1196] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/8250e1865dea45bfa18c6d5110a5bd9b
I20260927 18:38:18.602713 4142 ts_tablet_manager.cc:1939] T 8250e1865dea45bfa18c6d5110a5bd9b P c5d32c9b812c4e7f982c0c37f0d84977: Deleting consensus metadata
I20260927 18:38:18.603184 4140 ts_tablet_manager.cc:1939] T 8250e1865dea45bfa18c6d5110a5bd9b P a00e31f53d624485b5a62ef18cafa025: Deleting consensus metadata
I20260927 18:38:18.604077 3541 catalog_manager.cc:4841] TS c5d32c9b812c4e7f982c0c37f0d84977 (127.2.95.67:33637): tablet 8250e1865dea45bfa18c6d5110a5bd9b (table test-table [id=b2ab55f256d84df99ea0d5a4afa8f856]) successfully deleted
I20260927 18:38:18.604439 3540 catalog_manager.cc:4841] TS a00e31f53d624485b5a62ef18cafa025 (127.2.95.65:39593): tablet 8250e1865dea45bfa18c6d5110a5bd9b (table test-table [id=b2ab55f256d84df99ea0d5a4afa8f856]) successfully deleted
Sep 27 18:38:18 dist-test-slave-9jll krb5kdc[3511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534297, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:18.611761 4141 ts_tablet_manager.cc:1918] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:38:18.611943 4141 log.cc:1196] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/8250e1865dea45bfa18c6d5110a5bd9b
I20260927 18:38:18.612478 4141 ts_tablet_manager.cc:1939] T 8250e1865dea45bfa18c6d5110a5bd9b P a6c6499ad14d4283ba03a1b72491927d: Deleting consensus metadata
I20260927 18:38:18.614176 3542 catalog_manager.cc:4841] TS a6c6499ad14d4283ba03a1b72491927d (127.2.95.66:43655): tablet 8250e1865dea45bfa18c6d5110a5bd9b (table test-table [id=b2ab55f256d84df99ea0d5a4afa8f856]) successfully deleted
I20260927 18:38:18.621464 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.588989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.2.95.65:39593)
0927 18:38:18.589194 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.589202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.596801 (+ 7599us) server_negotiation.cc:407] Connection header received
0927 18:38:18.597398 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.597406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.597508 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.597594 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:18.599983 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.601204 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.603642 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.604332 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.604429 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.611063 (+ 6634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:18.611088 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:18.611116 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:18.611229 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:18.617115 (+ 5886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:18.618292 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:18.618305 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:18.618309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:18.618490 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:18.618924 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:18.618930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:18.618934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:18.619586 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:18.619970 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.620250 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.620534 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 18:38:18.622563 3758 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50632
I20260927 18:38:18.658870 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.626920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.2.95.126:45651)
0927 18:38:18.640010 (+ 13090us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.640018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.640708 (+ 690us) server_negotiation.cc:407] Connection header received
0927 18:38:18.640984 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.640990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.641113 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.641252 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:18.643438 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.644714 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.646873 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.647537 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.647687 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.651188 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:18.651196 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:18.651202 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:18.651268 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:18.655254 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:18.656246 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:18.656253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:18.656257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:18.656439 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:18.656881 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:18.656886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:18.656890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:18.657409 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:18.657770 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.658115 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.658397 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12319,"thread_start_us":12816,"threads_started":1}
W20260927 18:38:18.659711 3555 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34520
I20260927 18:38:18.663255 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 3627
I20260927 18:38:18.686743 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.666192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.2.95.65:39593)
0927 18:38:18.666437 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.666446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.666835 (+ 389us) server_negotiation.cc:407] Connection header received
0927 18:38:18.667082 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.667090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.667193 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.667318 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:18.669035 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.670391 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.673405 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.674207 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.674337 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.678331 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:18.678339 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:18.678345 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:18.678404 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:18.682840 (+ 4436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:18.683869 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:18.683877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:18.683881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:18.684074 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:18.684682 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:18.684688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:18.684692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:18.685123 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:18.685465 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.686024 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.686291 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 18:38:18.687575 3758 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50638
W20260927 18:38:18.688237 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 3627: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:38:18.688464 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 3627
I20260927 18:38:18.706723 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 3767
Sep 27 18:38:18 dist-test-slave-9jll krb5kdc[3511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534297, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:38:18.727423 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.707852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.2.95.66:43655)
0927 18:38:18.708037 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.708042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.708083 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:38:18.708147 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.708152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.708225 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.708304 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:18.710378 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.711353 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.713388 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.713877 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.713985 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.719512 (+ 5527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:18.719539 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:18.719565 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:18.719644 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:18.723914 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:18.724789 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:18.724801 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:18.724806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:18.724994 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:18.725403 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:18.725409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:18.725413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:18.726024 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:18.726395 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.726695 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.726973 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 18:38:18.728276 3898 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59834
W20260927 18:38:18.729063 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 3767: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:38:18.729254 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 3767
I20260927 18:38:18.749557 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 3907
Sep 27 18:38:18 dist-test-slave-9jll krb5kdc[3511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534297, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:38:18.770016 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:18.750498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.2.95.67:33637)
0927 18:38:18.750738 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:18.750745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:18.750776 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:38:18.750939 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:18.750946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:18.751065 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:18.751201 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:18.752676 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.753978 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.755490 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:18.756215 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:18.756372 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:18.761331 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:18.761360 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:18.761389 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:18.761451 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:18.766401 (+ 4950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:18.767399 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:18.767412 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:18.767416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:18.767627 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:18.768014 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:18.768023 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:18.768026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:18.768642 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:18.769033 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:18.769259 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 18:38:18.769523 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 18:38:18.770920 4038 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42796
W20260927 18:38:18.771806 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 3907: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:38:18.771974 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 3907
I20260927 18:38:18.791051 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 3524
I20260927 18:38:18.792433 3596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:18.919379 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 3524
2026-09-27T18:38:18Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5200 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:19 dist-test-slave-9jll krb5kdc[4152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:19 dist-test-slave-9jll krb5kdc[4152](info): set up 2 sockets
Sep 27 18:38:19 dist-test-slave-9jll krb5kdc[4152](info): commencing operation
krb5kdc: starting...
W20260927 18:38:19.939487 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:38:20 dist-test-slave-9jll krb5kdc[4152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:38:20.154178 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:45801
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:45801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:20.528214 4165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:20.528642 4165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:20.528781 4165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:20.536695 4165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:20.536835 4165 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:38:20.536881 4165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:20.536970 4165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:20.537050 4165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:20.537140 4165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:20.548151 4165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:45801
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:45801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:20.549916 4165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:20.551595 4165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:20.564653 4170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:20.564690 4171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:20.564860 4173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:20.565907 4165 server_base.cc:1034] running on GCE node
I20260927 18:38:20.566825 4165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:20.568574 4165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:20.569789 4165 hybrid_clock.cc:648] HybridClock initialized: now 1790534300569712 us; error 48 us; skew 500 ppm
I20260927 18:38:20.570173 4165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:20 dist-test-slave-9jll krb5kdc[4152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534300, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:20.573731 4165 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:38:20.575459 4165 webserver.cc:460] Webserver started at http://127.2.95.126:33861/ using document root <none> and password file <none>
I20260927 18:38:20.576295 4165 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:20.576397 4165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:20.576733 4165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:20.580252 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "c5041b3cdbd74a928c49fe81bbd645f3"
format_stamp: "Formatted at 2026-09-27 18:38:20 on dist-test-slave-9jll"
server_key: "485b7b6d6b03a68fc88709fa576c3a33"
server_key_iv: "1a2e2d3301633ae0e927b4befb7d85df"
server_key_version: "encryptionkey@0"
I20260927 18:38:20.581099 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "c5041b3cdbd74a928c49fe81bbd645f3"
format_stamp: "Formatted at 2026-09-27 18:38:20 on dist-test-slave-9jll"
server_key: "485b7b6d6b03a68fc88709fa576c3a33"
server_key_iv: "1a2e2d3301633ae0e927b4befb7d85df"
server_key_version: "encryptionkey@0"
I20260927 18:38:20.587070 4165 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 18:38:20.591148 4180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:20.592826 4165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:38:20.593003 4165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "c5041b3cdbd74a928c49fe81bbd645f3"
format_stamp: "Formatted at 2026-09-27 18:38:20 on dist-test-slave-9jll"
server_key: "485b7b6d6b03a68fc88709fa576c3a33"
server_key_iv: "1a2e2d3301633ae0e927b4befb7d85df"
server_key_version: "encryptionkey@0"
I20260927 18:38:20.593254 4165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:20.608515 4165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:20.613463 4165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:20.613796 4165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:20.637931 4165 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:45801
I20260927 18:38:20.638011 4241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:45801 every 8 connection(s)
I20260927 18:38:20.639564 4165 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:20.644330 4242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:20.646886 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 4165
I20260927 18:38:20.647094 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:20.647773 2429 external_mini_cluster.cc:1428] Setting key 6271514741298ca5e2ad23d07d461019
I20260927 18:38:20.657711 4242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3: Bootstrap starting.
I20260927 18:38:20.662282 4242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:20.663828 4242 log.cc:824] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:38:20 dist-test-slave-9jll krb5kdc[4152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:20.667315 4242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3: No bootstrap required, opened a new log
I20260927 18:38:20.673547 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:20.650320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.2.95.126:45801)
0927 18:38:20.651212 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:20.651227 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:20.651278 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:38:20.653183 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:20.653213 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:20.653781 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:20.654229 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:20.656178 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:20.657659 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:20.659466 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:20.660300 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:20.660458 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:20.665625 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:20.665635 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:20.665654 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:20.665728 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:20.668932 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:20.669750 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:20.669761 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:20.669769 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:20.669960 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:20.670349 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:20.670353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:20.670355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:20.670980 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:20.671300 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:20.671790 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 18:38:20.672295 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":387,"threads_started":1}
I20260927 18:38:20.675918 4242 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5041b3cdbd74a928c49fe81bbd645f3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45801 } }
I20260927 18:38:20.676306 4242 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:20.676414 4242 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5041b3cdbd74a928c49fe81bbd645f3, State: Initialized, Role: FOLLOWER
I20260927 18:38:20.677094 4242 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5041b3cdbd74a928c49fe81bbd645f3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45801 } }
I20260927 18:38:20.677335 4242 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:20.677459 4242 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:20.677618 4242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:20.679982 4242 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5041b3cdbd74a928c49fe81bbd645f3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45801 } }
I20260927 18:38:20.680653 4242 leader_election.cc:302] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5041b3cdbd74a928c49fe81bbd645f3; no voters:
I20260927 18:38:20.681507 4242 leader_election.cc:288] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:20.681843 4247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:20.682981 4247 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [term 1 LEADER]: Becoming Leader. State: Replica: c5041b3cdbd74a928c49fe81bbd645f3, State: Running, Role: LEADER
I20260927 18:38:20.683653 4247 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5041b3cdbd74a928c49fe81bbd645f3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45801 } }
I20260927 18:38:20.684561 4242 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:20.688295 4249 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5041b3cdbd74a928c49fe81bbd645f3. Latest consensus state: current_term: 1 leader_uuid: "c5041b3cdbd74a928c49fe81bbd645f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5041b3cdbd74a928c49fe81bbd645f3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45801 } } }
I20260927 18:38:20.688650 4249 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:20.690714 4254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:20.690678 4248 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5041b3cdbd74a928c49fe81bbd645f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5041b3cdbd74a928c49fe81bbd645f3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45801 } } }
I20260927 18:38:20.691025 4248 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:20.697722 4254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:20.712414 4254 catalog_manager.cc:1347] Generated new cluster ID: 835bb2a4e1634376813ffed2a2a8e139
I20260927 18:38:20.712577 4254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:20.721330 4254 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:20.723552 4254 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:20.734305 4254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3: Generated new TSK 0
I20260927 18:38:20.735716 4254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:38:20.795282 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:45801
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:21.185230 4268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:21.185616 4268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:21.185804 4268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:21.194453 4268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:21.194620 4268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:21.194754 4268 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:38:21.194826 4268 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:38:21.194943 4268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:38:21.206143 4268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.95.126:45801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:21.207954 4268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:21.209580 4268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:21.224493 4276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:21.224511 4273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:21.224905 4274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:21.225955 4268 server_base.cc:1034] running on GCE node
I20260927 18:38:21.226701 4268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:21.227850 4268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:21.229054 4268 hybrid_clock.cc:648] HybridClock initialized: now 1790534301228994 us; error 58 us; skew 500 ppm
I20260927 18:38:21.229417 4268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:21 dist-test-slave-9jll krb5kdc[4152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:21.233085 4268 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:38:21.235033 4268 webserver.cc:460] Webserver started at http://127.2.95.65:36267/ using document root <none> and password file <none>
I20260927 18:38:21.235888 4268 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:21.236016 4268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:21.236366 4268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:21.239840 4268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "5b8fc5f189944a62ba5a60aba519ed25"
format_stamp: "Formatted at 2026-09-27 18:38:21 on dist-test-slave-9jll"
server_key: "9e74d1bd2434af17f9b6fa393983cc4f"
server_key_iv: "cd70da705a4f2419e83b33a39fd5927f"
server_key_version: "encryptionkey@0"
I20260927 18:38:21.240707 4268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "5b8fc5f189944a62ba5a60aba519ed25"
format_stamp: "Formatted at 2026-09-27 18:38:21 on dist-test-slave-9jll"
server_key: "9e74d1bd2434af17f9b6fa393983cc4f"
server_key_iv: "cd70da705a4f2419e83b33a39fd5927f"
server_key_version: "encryptionkey@0"
I20260927 18:38:21.246871 4268 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 18:38:21.251040 4283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:21.252686 4268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260927 18:38:21.252858 4268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "5b8fc5f189944a62ba5a60aba519ed25"
format_stamp: "Formatted at 2026-09-27 18:38:21 on dist-test-slave-9jll"
server_key: "9e74d1bd2434af17f9b6fa393983cc4f"
server_key_iv: "cd70da705a4f2419e83b33a39fd5927f"
server_key_version: "encryptionkey@0"
I20260927 18:38:21.253079 4268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:21.297122 4268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:21.301862 4268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:21.302191 4268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:21.305200 4268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:21.305307 4268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:21.305421 4268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:21.305495 4268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:38:21 dist-test-slave-9jll krb5kdc[4152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:21.330893 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:21.306108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:59849 (local address 127.2.95.126:45801)
0927 18:38:21.306610 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:21.306617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:21.308149 (+ 1532us) server_negotiation.cc:407] Connection header received
0927 18:38:21.311024 (+ 2875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:21.311031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:21.311148 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:21.311278 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:21.313748 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.314798 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.317017 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.317590 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.317779 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:21.323248 (+ 5469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:21.323257 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:21.323264 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:21.323317 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:21.327073 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:21.327890 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:21.327897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:21.327901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:21.328076 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:21.328525 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:21.328529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:21.328533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:21.328963 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:21.329127 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:21.330217 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0927 18:38:21.330490 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":232,"threads_started":1}
I20260927 18:38:21.332796 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:21.306957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45801 (local address 127.2.95.65:59849)
0927 18:38:21.307925 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:21.307996 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:21.310595 (+ 2599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:21.311643 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:21.311660 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:21.312419 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:21.313522 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:21.313544 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.315041 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.315045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:21.316838 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:21.316852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.317920 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.317929 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:21.318318 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:21.319312 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:21.319341 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:21.322854 (+ 3513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:21.327253 (+ 4399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:21.327266 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:21.327287 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:21.327772 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:21.328193 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:21.328197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:21.328199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:21.328433 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:21.329141 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:21.329148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:21.329623 (+ 475us) client_negotiation.cc:769] Sending connection context
0927 18:38:21.330016 (+ 393us) client_negotiation.cc:240] Negotiation successful
0927 18:38:21.330721 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":419,"threads_started":1}
I20260927 18:38:21.350844 4268 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:34915
I20260927 18:38:21.350862 4403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:34915 every 8 connection(s)
I20260927 18:38:21.352614 4268 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:21.355389 4404 heartbeater.cc:347] Connected to a master server at 127.2.95.126:45801
I20260927 18:38:21.355741 4404 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:21.356271 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:21.341486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45801 (local address 127.2.95.65:38269)
0927 18:38:21.342189 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:21.342210 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:21.342404 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:21.342769 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:21.342778 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:21.343478 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:21.343949 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:21.343965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.345345 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.345349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:21.347053 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:21.347067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.348248 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.348253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:21.348386 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:21.349160 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:21.349179 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:21.350240 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:21.353396 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:21.353402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:21.353407 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:21.353912 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:21.354292 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:21.354297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:21.354301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:21.354438 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:21.355183 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:21.355187 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:21.355409 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 18:38:21.355525 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 18:38:21.355889 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":424,"threads_started":1}
I20260927 18:38:21.357163 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:21.341453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:38269 (local address 127.2.95.126:45801)
0927 18:38:21.341689 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:21.341696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:21.342329 (+ 633us) server_negotiation.cc:407] Connection header received
0927 18:38:21.342548 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:21.342555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:21.342637 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:21.342725 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:21.344176 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.345199 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.347196 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.347812 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.347951 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:21.350438 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:21.350443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:21.350447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:21.350482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:21.353268 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:21.354039 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:21.354043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:21.354046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:21.354185 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:21.354542 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:21.354545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:21.354548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:21.354872 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:21.355033 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:21.356600 (+ 1567us) server_negotiation.cc:299] Negotiation successful
0927 18:38:21.356839 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 18:38:21.357158 4404 heartbeater.cc:511] Master 127.2.95.126:45801 requested a full tablet report, sending...
I20260927 18:38:21.360225 4197 ts_manager.cc:194] Registered new tserver with Master: 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65:34915)
I20260927 18:38:21.362165 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 4268
I20260927 18:38:21.362215 4197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:59849
I20260927 18:38:21.362390 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:21.363106 2429 external_mini_cluster.cc:1428] Setting key b45efb970e1e853dd39cd01313a9e665
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:38:21.407016 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:45801
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:21.782291 4408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:21.782646 4408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:21.782758 4408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:21.790678 4408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:21.790815 4408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:21.790923 4408 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:38:21.790979 4408 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:38:21.791046 4408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:38:21.802004 4408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.95.126:45801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:21.803718 4408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:21.805311 4408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:21.821226 4416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:21.821256 4414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:21.821292 4413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:21.822552 4408 server_base.cc:1034] running on GCE node
I20260927 18:38:21.823355 4408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:21.824571 4408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:21.825835 4408 hybrid_clock.cc:648] HybridClock initialized: now 1790534301825752 us; error 63 us; skew 500 ppm
I20260927 18:38:21.826346 4408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:21 dist-test-slave-9jll krb5kdc[4152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:21.830266 4408 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:38:21.832324 4408 webserver.cc:460] Webserver started at http://127.2.95.66:45597/ using document root <none> and password file <none>
I20260927 18:38:21.833209 4408 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:21.833351 4408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:21.833756 4408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:21.837518 4408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "353d30df674c43f09eb1d1a19f759061"
format_stamp: "Formatted at 2026-09-27 18:38:21 on dist-test-slave-9jll"
server_key: "f3de7b222c7919fac07d4688103f2a15"
server_key_iv: "1603953f81a4ee32795594c2b3a39c86"
server_key_version: "encryptionkey@0"
I20260927 18:38:21.838465 4408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "353d30df674c43f09eb1d1a19f759061"
format_stamp: "Formatted at 2026-09-27 18:38:21 on dist-test-slave-9jll"
server_key: "f3de7b222c7919fac07d4688103f2a15"
server_key_iv: "1603953f81a4ee32795594c2b3a39c86"
server_key_version: "encryptionkey@0"
I20260927 18:38:21.844877 4408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 18:38:21.849095 4423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:21.850804 4408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:38:21.850988 4408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "353d30df674c43f09eb1d1a19f759061"
format_stamp: "Formatted at 2026-09-27 18:38:21 on dist-test-slave-9jll"
server_key: "f3de7b222c7919fac07d4688103f2a15"
server_key_iv: "1603953f81a4ee32795594c2b3a39c86"
server_key_version: "encryptionkey@0"
I20260927 18:38:21.851284 4408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:21.886296 4408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:21.891561 4408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:21.891883 4408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:21.895262 4408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:21.895396 4408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:21.895524 4408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:21.895599 4408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:38:21 dist-test-slave-9jll krb5kdc[4152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534301, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:21.924762 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:21.896279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:58293 (local address 127.2.95.126:45801)
0927 18:38:21.896795 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:21.896799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:21.898579 (+ 1780us) server_negotiation.cc:407] Connection header received
0927 18:38:21.900842 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:21.900848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:21.900973 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:21.901112 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:21.904565 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.905718 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.909071 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.909762 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.909919 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:21.915788 (+ 5869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:21.915794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:21.915798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:21.915840 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:21.919864 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:21.921451 (+ 1587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:21.921459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:21.921462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:21.921700 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:21.922423 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:21.922429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:21.922433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:21.922887 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:21.923132 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:21.924040 (+ 908us) server_negotiation.cc:299] Negotiation successful
0927 18:38:21.924335 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":281,"threads_started":1}
I20260927 18:38:21.926482 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:21.897142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45801 (local address 127.2.95.66:58293)
0927 18:38:21.898260 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:21.898314 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:21.900484 (+ 2170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:21.901494 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:21.901510 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:21.902862 (+ 1352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:21.904181 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:21.904223 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.905966 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.905973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:21.908765 (+ 2792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:21.908785 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.910099 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.910107 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:21.910510 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:21.911487 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:21.911534 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:21.915414 (+ 3880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:21.920223 (+ 4809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:21.920242 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:21.920277 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:21.921229 (+ 952us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:21.921857 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:21.921864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:21.921867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:21.922276 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:21.923053 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:21.923068 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:21.923597 (+ 529us) client_negotiation.cc:769] Sending connection context
0927 18:38:21.923885 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 18:38:21.924587 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":519,"threads_started":1}
I20260927 18:38:21.939195 4408 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:45243
I20260927 18:38:21.939240 4542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:45243 every 8 connection(s)
I20260927 18:38:21.940869 4408 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:21.943337 4544 heartbeater.cc:347] Connected to a master server at 127.2.95.126:45801
I20260927 18:38:21.943625 4544 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:21.944689 4544 heartbeater.cc:511] Master 127.2.95.126:45801 requested a full tablet report, sending...
I20260927 18:38:21.946687 4197 ts_manager.cc:194] Registered new tserver with Master: 353d30df674c43f09eb1d1a19f759061 (127.2.95.66:45243)
I20260927 18:38:21.947961 4197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:58293
I20260927 18:38:21.950083 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 4408
I20260927 18:38:21.950286 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:21.950789 2429 external_mini_cluster.cc:1428] Setting key d9f45108065333d0ea576ca23a15003f
I20260927 18:38:21.954396 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:21.940476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:60959 (local address 127.2.95.126:45801)
0927 18:38:21.940704 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:21.940711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:21.941262 (+ 551us) server_negotiation.cc:407] Connection header received
0927 18:38:21.941451 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:21.941457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:21.941536 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:21.941623 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:21.942778 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.943820 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.945320 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.945876 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.946008 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:21.947987 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:21.947992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:21.947997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:21.948029 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:21.951551 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:21.952310 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:21.952317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:21.952320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:21.952505 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:21.952962 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:21.952967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:21.952971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:21.953314 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:21.953432 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:21.953937 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 18:38:21.954142 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:38:21.954715 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:21.940395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45801 (local address 127.2.95.66:60959)
0927 18:38:21.941141 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:21.941164 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:21.941308 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:21.941699 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:21.941703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:21.942225 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:21.942638 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:21.942654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.943949 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.943958 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:21.945186 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:21.945199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:21.946062 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:21.946069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:21.946246 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:21.946951 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:21.946968 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:21.947774 (+ 806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:21.951763 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:21.951768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:21.951772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:21.952191 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:21.952664 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:21.952669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:21.952673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:21.952847 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:21.953438 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:21.953443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:21.953647 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 18:38:21.953857 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 18:38:21.954260 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":402,"threads_started":1}
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:38:21.998790 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:45801
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:38:22.365967 4404 heartbeater.cc:503] Master 127.2.95.126:45801 was elected leader, sending a full tablet report...
W20260927 18:38:22.398342 4548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:22.398797 4548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:22.398988 4548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:22.407126 4548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:22.407297 4548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:22.407431 4548 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 18:38:22.407501 4548 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:38:22.407612 4548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:38:22.418533 4548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.95.126:45801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:22.420287 4548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:22.422092 4548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:22.437821 4554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:22.437867 4556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:22.437846 4553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:22.438772 4548 server_base.cc:1034] running on GCE node
I20260927 18:38:22.439594 4548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:22.440814 4548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:22.442055 4548 hybrid_clock.cc:648] HybridClock initialized: now 1790534302441984 us; error 73 us; skew 500 ppm
I20260927 18:38:22.442466 4548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:22 dist-test-slave-9jll krb5kdc[4152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534302, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:22.451800 4548 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:38:22.453845 4548 webserver.cc:460] Webserver started at http://127.2.95.67:42457/ using document root <none> and password file <none>
I20260927 18:38:22.454721 4548 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:22.454866 4548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:22.455278 4548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:22.459263 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3"
format_stamp: "Formatted at 2026-09-27 18:38:22 on dist-test-slave-9jll"
server_key: "9bf0ccd9e388e6dabd1ef6e9bd03a3ae"
server_key_iv: "e178f512d6507419adb86807fd55808d"
server_key_version: "encryptionkey@0"
I20260927 18:38:22.460287 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3"
format_stamp: "Formatted at 2026-09-27 18:38:22 on dist-test-slave-9jll"
server_key: "9bf0ccd9e388e6dabd1ef6e9bd03a3ae"
server_key_iv: "e178f512d6507419adb86807fd55808d"
server_key_version: "encryptionkey@0"
I20260927 18:38:22.467294 4548 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260927 18:38:22.471468 4563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:22.473161 4548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:38:22.473357 4548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3"
format_stamp: "Formatted at 2026-09-27 18:38:22 on dist-test-slave-9jll"
server_key: "9bf0ccd9e388e6dabd1ef6e9bd03a3ae"
server_key_iv: "e178f512d6507419adb86807fd55808d"
server_key_version: "encryptionkey@0"
I20260927 18:38:22.473594 4548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:22.506157 4548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:22.511624 4548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:22.511984 4548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:22.515268 4548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:22.515399 4548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:22.515512 4548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:22.515609 4548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:38:22 dist-test-slave-9jll krb5kdc[4152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534302, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:22.544128 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.516239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:40807 (local address 127.2.95.126:45801)
0927 18:38:22.516802 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:22.516808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:22.518412 (+ 1604us) server_negotiation.cc:407] Connection header received
0927 18:38:22.520959 (+ 2547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:22.520968 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:22.521075 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:22.521214 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:22.524246 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.525259 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.528274 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.528873 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.529029 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.535204 (+ 6175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:22.535213 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:22.535219 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:22.535276 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:22.539428 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:22.540885 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:22.540892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:22.540895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:22.541068 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:22.541788 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:22.541794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:22.541798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:22.542195 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:22.542369 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:22.543580 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0927 18:38:22.543800 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":256,"threads_started":1}
I20260927 18:38:22.546316 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.517121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45801 (local address 127.2.95.67:40807)
0927 18:38:22.518211 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:22.518260 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:22.520424 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:22.521590 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:22.521606 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:22.522704 (+ 1098us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:22.523929 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.523966 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.525504 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.525510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.528078 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.528102 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.529217 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.529227 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.529558 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.530611 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:22.530655 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:22.534726 (+ 4071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:22.539648 (+ 4922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:22.539662 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:22.539714 (+ 52us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:22.540678 (+ 964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:22.541215 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:22.541222 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:22.541226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:22.541616 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:22.542365 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:22.542378 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:22.543058 (+ 680us) client_negotiation.cc:769] Sending connection context
0927 18:38:22.543460 (+ 402us) client_negotiation.cc:240] Negotiation successful
0927 18:38:22.544234 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":472,"threads_started":1}
I20260927 18:38:22.559981 4548 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:40655
I20260927 18:38:22.560096 4683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:40655 every 8 connection(s)
I20260927 18:38:22.561630 4548 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:38:22.562346 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 4548
I20260927 18:38:22.562559 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:22.563201 2429 external_mini_cluster.cc:1428] Setting key b1dae6f3c9a2ccf09734dcc397298984
I20260927 18:38:22.564287 4684 heartbeater.cc:347] Connected to a master server at 127.2.95.126:45801
I20260927 18:38:22.564610 4684 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:22.566143 4684 heartbeater.cc:511] Master 127.2.95.126:45801 requested a full tablet report, sending...
I20260927 18:38:22.567940 4197 ts_manager.cc:194] Registered new tserver with Master: 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655)
I20260927 18:38:22.569036 4197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:40807
I20260927 18:38:22.577045 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.554750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45801 (local address 127.2.95.67:37691)
0927 18:38:22.562275 (+ 7525us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:22.562313 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:22.562709 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:22.563461 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:22.563467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:22.564431 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:22.564924 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.564940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.566845 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.566852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.568406 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.568424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.569536 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.569545 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.569757 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.570433 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:22.570457 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:22.571579 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:22.574718 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:22.574722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:22.574725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:22.575179 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:22.575529 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:22.575532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:22.575534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:22.575658 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:22.576142 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:22.576146 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:22.576300 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 18:38:22.576419 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 18:38:22.576660 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7311,"thread_start_us":7151,"threads_started":1}
I20260927 18:38:22.577001 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.554699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:37691 (local address 127.2.95.126:45801)
0927 18:38:22.554963 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:22.554970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:22.562490 (+ 7520us) server_negotiation.cc:407] Connection header received
0927 18:38:22.562921 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:22.562930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:22.563059 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:22.563158 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:22.565191 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.566556 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.568585 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.569175 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.569310 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.571776 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:22.571782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:22.571786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:22.571822 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:22.574589 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:22.575297 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:22.575301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:22.575304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:22.575447 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:22.575736 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:22.575739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:22.575741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:22.576024 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:22.576132 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:22.576547 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 18:38:22.576742 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":78}
I20260927 18:38:22.579641 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:38:22 dist-test-slave-9jll krb5kdc[4152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:38:22 dist-test-slave-9jll krb5kdc[4152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:22.633369 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.617511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.2.95.126:45801)
0927 18:38:22.617818 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:22.617826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:22.618172 (+ 346us) server_negotiation.cc:407] Connection header received
0927 18:38:22.618379 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:22.618383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:22.618469 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:22.618586 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:22.620251 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.621084 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.622468 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.622921 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.623024 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.627378 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:22.627384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:22.627387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:22.627433 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:22.630436 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:22.631207 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:22.631212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:22.631215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:22.631356 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:22.631809 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:22.631814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:22.631816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:22.632146 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:22.632363 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:22.632743 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 18:38:22.632940 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 18:38:22.639165 4197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:38:22.651031 4197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:22.679565 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.666165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:40655 (local address 127.0.0.1:37010)
0927 18:38:22.667423 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:22.667479 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:22.667704 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:22.669251 (+ 1547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:22.669261 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:22.669300 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:22.669857 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.669873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.672674 (+ 2801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.672681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.675509 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.675532 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.678236 (+ 2704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.678244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.678589 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.678675 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 18:38:22.678824 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 18:38:22.679107 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"threads_started":1}
I20260927 18:38:22.681097 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.666556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.2.95.67:40655)
0927 18:38:22.668444 (+ 1888us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:22.668456 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:22.668496 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:38:22.668634 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:22.668643 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:22.668958 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:22.670128 (+ 1170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:22.670507 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.672480 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.675804 (+ 3324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.677921 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.680190 (+ 2269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.680472 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:22.680563 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 18:38:22.680742 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1696,"thread_start_us":1479,"threads_started":1}
I20260927 18:38:22.687791 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.667324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.2.95.65:34915)
0927 18:38:22.680481 (+ 13157us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:22.680503 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:22.680540 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:38:22.680667 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:22.680677 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:22.681004 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:22.682185 (+ 1181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:22.682425 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.683863 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.685773 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.686825 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.686944 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.687192 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:22.687338 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 18:38:22.687510 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12966,"thread_start_us":14060,"threads_started":1}
I20260927 18:38:22.687783 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.666522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34915 (local address 127.0.0.1:49580)
0927 18:38:22.679865 (+ 13343us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:22.679887 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:22.680020 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:22.681764 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:22.681769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:22.681796 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:22.681996 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.682009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.684000 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.684004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.685570 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.685584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.686985 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.686990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.687163 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.687196 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:38:22.687284 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 18:38:22.687463 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13213,"thread_start_us":636,"threads_started":1}
I20260927 18:38:22.687345 4614 tablet_service.cc:1467] Processing CreateTablet for tablet e953c64d22a64992ae1a700311708c3b (DEFAULT_TABLE table=test-table [id=bd194e8c4187478c9ebc4deffe129e05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:22.689389 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e953c64d22a64992ae1a700311708c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:22.691895 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.668305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:45243 (local address 127.0.0.1:57568)
0927 18:38:22.680593 (+ 12288us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:22.680621 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:22.680760 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:22.683071 (+ 2311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:22.683077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:22.683104 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:22.683466 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.683477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.685276 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.685281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.687373 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.687387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.689354 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.689360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.689555 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.689588 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:38:22.690632 (+ 1044us) client_negotiation.cc:240] Negotiation successful
0927 18:38:22.690791 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12141,"thread_start_us":14502,"threads_started":1}
I20260927 18:38:22.693408 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.668924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.2.95.66:45243)
0927 18:38:22.681864 (+ 12940us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:22.681873 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:22.681917 (+ 44us) server_negotiation.cc:407] Connection header received
0927 18:38:22.682048 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:22.682058 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:22.682312 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:22.682526 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:22.683655 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.685099 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.687563 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.688885 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.689004 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.689204 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:22.691222 (+ 2018us) server_negotiation.cc:299] Negotiation successful
0927 18:38:22.692954 (+ 1732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12769,"thread_start_us":12604,"threads_started":1}
I20260927 18:38:22.694093 4334 tablet_service.cc:1467] Processing CreateTablet for tablet e953c64d22a64992ae1a700311708c3b (DEFAULT_TABLE table=test-table [id=bd194e8c4187478c9ebc4deffe129e05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:22.695779 4334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e953c64d22a64992ae1a700311708c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:22.696863 4474 tablet_service.cc:1467] Processing CreateTablet for tablet e953c64d22a64992ae1a700311708c3b (DEFAULT_TABLE table=test-table [id=bd194e8c4187478c9ebc4deffe129e05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:22.698239 4474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e953c64d22a64992ae1a700311708c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:22.707480 4703 tablet_bootstrap.cc:492] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Bootstrap starting.
I20260927 18:38:22.711130 4704 tablet_bootstrap.cc:492] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25: Bootstrap starting.
I20260927 18:38:22.712826 4703 tablet_bootstrap.cc:654] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:22.714634 4704 tablet_bootstrap.cc:654] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:22.715473 4703 log.cc:824] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:22.715654 4704 log.cc:824] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:22.717715 4705 tablet_bootstrap.cc:492] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061: Bootstrap starting.
I20260927 18:38:22.718879 4704 tablet_bootstrap.cc:492] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25: No bootstrap required, opened a new log
I20260927 18:38:22.719157 4704 ts_tablet_manager.cc:1397] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 18:38:22.722072 4705 tablet_bootstrap.cc:654] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:22.723546 4705 log.cc:824] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:22.725807 4703 tablet_bootstrap.cc:492] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: No bootstrap required, opened a new log
I20260927 18:38:22.726161 4703 ts_tablet_manager.cc:1397] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20260927 18:38:22.726765 4705 tablet_bootstrap.cc:492] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061: No bootstrap required, opened a new log
I20260927 18:38:22.727053 4705 ts_tablet_manager.cc:1397] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260927 18:38:22.727593 4704 raft_consensus.cc:348] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } }
I20260927 18:38:22.727991 4704 raft_consensus.cc:374] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:22.728096 4704 raft_consensus.cc:729] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8fc5f189944a62ba5a60aba519ed25, State: Initialized, Role: FOLLOWER
I20260927 18:38:22.728747 4704 consensus_queue.cc:260] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } }
I20260927 18:38:22.730679 4704 ts_tablet_manager.cc:1428] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 18:38:22.737720 4703 raft_consensus.cc:348] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } }
I20260927 18:38:22.737721 4705 raft_consensus.cc:348] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } }
I20260927 18:38:22.738314 4703 raft_consensus.cc:374] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:22.738314 4705 raft_consensus.cc:374] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:22.738427 4703 raft_consensus.cc:729] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd4ea9d66c843fcb607f0a73ba5b3a3, State: Initialized, Role: FOLLOWER
I20260927 18:38:22.738427 4705 raft_consensus.cc:729] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353d30df674c43f09eb1d1a19f759061, State: Initialized, Role: FOLLOWER
I20260927 18:38:22.739200 4705 consensus_queue.cc:260] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } }
I20260927 18:38:22.739200 4703 consensus_queue.cc:260] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } }
I20260927 18:38:22.740487 4544 heartbeater.cc:503] Master 127.2.95.126:45801 was elected leader, sending a full tablet report...
I20260927 18:38:22.741303 4703 ts_tablet_manager.cc:1428] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260927 18:38:22.741838 4684 heartbeater.cc:503] Master 127.2.95.126:45801 was elected leader, sending a full tablet report...
I20260927 18:38:22.742677 4705 ts_tablet_manager.cc:1428] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
W20260927 18:38:22.818130 4685 tablet.cc:2369] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:38:22.861644 4405 tablet.cc:2369] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:38:22.901698 4710 raft_consensus.cc:482] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:22.902145 4710 raft_consensus.cc:504] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } }
I20260927 18:38:22.904342 4710 leader_election.cc:288] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655), 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65:34915)
I20260927 18:38:22.911773 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.904974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:51365 (local address 127.2.95.65:34915)
0927 18:38:22.905195 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:22.905203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:22.906047 (+ 844us) server_negotiation.cc:407] Connection header received
0927 18:38:22.906269 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:22.906273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:22.906354 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:22.906435 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:22.907135 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.907885 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.909959 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.910784 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.910911 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.911019 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:22.911337 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 18:38:22.911477 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:38:22.911772 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.904781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:46305 (local address 127.2.95.67:40655)
0927 18:38:22.905030 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:22.905037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:22.906115 (+ 1078us) server_negotiation.cc:407] Connection header received
0927 18:38:22.906282 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:22.906288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:22.906354 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:22.906435 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:22.907104 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.907895 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.909953 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.910750 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.910869 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.911002 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:22.911344 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 18:38:22.911482 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:38:22.911970 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.904810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:40655 (local address 127.2.95.66:46305)
0927 18:38:22.905988 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:22.906003 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:22.906137 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:22.906458 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:22.906462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:22.906493 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:22.906938 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.906948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.908015 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.908021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.909808 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.909821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.911003 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.911009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.911158 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.911209 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 18:38:22.911293 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:38:22.911497 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"mutex_wait_us":149,"thread_start_us":615,"threads_started":1}
I20260927 18:38:22.912020 4712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:22.904799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34915 (local address 127.2.95.66:51365)
0927 18:38:22.905908 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:22.905927 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:22.906115 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:22.906462 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:22.906466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:22.906561 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:22.906992 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.907001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.908018 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.908023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.909816 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:22.909827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:22.911012 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:22.911017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:22.911172 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:22.911214 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 18:38:22.911293 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:38:22.911487 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":121,"spinlock_wait_cycles":33024,"thread_start_us":657,"threads_started":1}
I20260927 18:38:22.913539 4354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e953c64d22a64992ae1a700311708c3b" candidate_uuid: "353d30df674c43f09eb1d1a19f759061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8fc5f189944a62ba5a60aba519ed25" is_pre_election: true
I20260927 18:38:22.913535 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e953c64d22a64992ae1a700311708c3b" candidate_uuid: "353d30df674c43f09eb1d1a19f759061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" is_pre_election: true
I20260927 18:38:22.914281 4354 raft_consensus.cc:2454] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353d30df674c43f09eb1d1a19f759061 in term 0.
I20260927 18:38:22.914280 4634 raft_consensus.cc:2454] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353d30df674c43f09eb1d1a19f759061 in term 0.
I20260927 18:38:22.915124 4424 leader_election.cc:302] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353d30df674c43f09eb1d1a19f759061, 5b8fc5f189944a62ba5a60aba519ed25; no voters:
I20260927 18:38:22.915565 4710 raft_consensus.cc:2781] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:22.915722 4710 raft_consensus.cc:482] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:22.915822 4710 raft_consensus.cc:3037] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:22.918391 4710 raft_consensus.cc:504] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } }
I20260927 18:38:22.919085 4710 leader_election.cc:288] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [CANDIDATE]: Term 1 election: Requested vote from peers 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655), 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65:34915)
I20260927 18:38:22.919770 4354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e953c64d22a64992ae1a700311708c3b" candidate_uuid: "353d30df674c43f09eb1d1a19f759061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8fc5f189944a62ba5a60aba519ed25"
I20260927 18:38:22.919770 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e953c64d22a64992ae1a700311708c3b" candidate_uuid: "353d30df674c43f09eb1d1a19f759061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3"
I20260927 18:38:22.920022 4634 raft_consensus.cc:3037] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:22.920104 4354 raft_consensus.cc:3037] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:22.923892 4354 raft_consensus.cc:2454] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353d30df674c43f09eb1d1a19f759061 in term 1.
I20260927 18:38:22.923892 4634 raft_consensus.cc:2454] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353d30df674c43f09eb1d1a19f759061 in term 1.
I20260927 18:38:22.924420 4424 leader_election.cc:302] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2dd4ea9d66c843fcb607f0a73ba5b3a3, 353d30df674c43f09eb1d1a19f759061, 5b8fc5f189944a62ba5a60aba519ed25; no voters:
I20260927 18:38:22.924803 4710 raft_consensus.cc:2781] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:22.925328 4710 raft_consensus.cc:686] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 1 LEADER]: Becoming Leader. State: Replica: 353d30df674c43f09eb1d1a19f759061, State: Running, Role: LEADER
I20260927 18:38:22.925943 4710 consensus_queue.cc:237] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } }
I20260927 18:38:22.931725 4196 catalog_manager.cc:5495] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 reported cstate change: term changed from 0 to 1, leader changed from <none> to 353d30df674c43f09eb1d1a19f759061 (127.2.95.66). New cstate: current_term: 1 leader_uuid: "353d30df674c43f09eb1d1a19f759061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } health_report { overall_health: HEALTHY } } }
W20260927 18:38:22.949359 4545 tablet.cc:2369] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 18:38:23 dist-test-slave-9jll krb5kdc[4152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534300, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:23.024384 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.008828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45801 (local address 127.0.0.1:43556)
0927 18:38:23.009586 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:23.009604 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:23.009846 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:23.010205 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:23.010209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:23.011061 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:23.011369 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.011379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.012592 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.012596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.013723 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.013737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.014641 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.014647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.014789 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.015559 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:23.015578 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:23.018346 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:23.021626 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:23.021641 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:23.021649 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:23.022251 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:23.022675 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:23.022679 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:23.022682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:23.022824 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:23.023411 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:23.023419 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:23.023605 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 18:38:23.023740 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 18:38:23.024009 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":458,"threads_started":1}
I20260927 18:38:23.024466 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.008828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.2.95.126:45801)
0927 18:38:23.009095 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.009101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.009785 (+ 684us) server_negotiation.cc:407] Connection header received
0927 18:38:23.009969 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.009973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.010044 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.010150 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:23.011550 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.012427 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.013875 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.014407 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.014544 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.018554 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:23.018560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:23.018564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:23.018602 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:23.021461 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:23.022395 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:23.022400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:23.022403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:23.022553 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:23.022956 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:23.022961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:23.022963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:23.023287 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:23.023412 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.023872 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.024197 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":76}
I20260927 18:38:23.030337 4196 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:43556:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:38:23.031612 4196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:23.039451 4474 tablet_service.cc:1467] Processing CreateTablet for tablet d1db65b0021a464081f47ae0b8d4f15b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a799eb8e043d49e7904883f6e8dfc0ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:38:23.039736 4614 tablet_service.cc:1467] Processing CreateTablet for tablet d1db65b0021a464081f47ae0b8d4f15b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a799eb8e043d49e7904883f6e8dfc0ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:38:23.039379 4334 tablet_service.cc:1467] Processing CreateTablet for tablet d1db65b0021a464081f47ae0b8d4f15b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a799eb8e043d49e7904883f6e8dfc0ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:38:23.040572 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1db65b0021a464081f47ae0b8d4f15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:23.040583 4334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1db65b0021a464081f47ae0b8d4f15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:23.040570 4474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1db65b0021a464081f47ae0b8d4f15b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:23.047107 4705 tablet_bootstrap.cc:492] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061: Bootstrap starting.
I20260927 18:38:23.048089 4704 tablet_bootstrap.cc:492] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: Bootstrap starting.
I20260927 18:38:23.048301 4703 tablet_bootstrap.cc:492] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Bootstrap starting.
I20260927 18:38:23.049016 4705 tablet_bootstrap.cc:654] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:23.049731 4703 tablet_bootstrap.cc:654] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:23.049885 4704 tablet_bootstrap.cc:654] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:23.052729 4704 tablet_bootstrap.cc:492] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: No bootstrap required, opened a new log
I20260927 18:38:23.052829 4705 tablet_bootstrap.cc:492] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061: No bootstrap required, opened a new log
I20260927 18:38:23.052902 4704 ts_tablet_manager.cc:1397] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 18:38:23.052979 4705 ts_tablet_manager.cc:1397] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20260927 18:38:23.053364 4703 tablet_bootstrap.cc:492] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: No bootstrap required, opened a new log
I20260927 18:38:23.053519 4703 ts_tablet_manager.cc:1397] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:38:23.053799 4705 raft_consensus.cc:348] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.053828 4704 raft_consensus.cc:348] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.054023 4705 raft_consensus.cc:374] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:23.054082 4704 raft_consensus.cc:374] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:23.054117 4705 raft_consensus.cc:729] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353d30df674c43f09eb1d1a19f759061, State: Initialized, Role: FOLLOWER
I20260927 18:38:23.054203 4704 raft_consensus.cc:729] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8fc5f189944a62ba5a60aba519ed25, State: Initialized, Role: FOLLOWER
I20260927 18:38:23.054448 4705 consensus_queue.cc:260] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.054404 4703 raft_consensus.cc:348] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.054453 4704 consensus_queue.cc:260] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.054677 4703 raft_consensus.cc:374] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:23.054775 4703 raft_consensus.cc:729] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd4ea9d66c843fcb607f0a73ba5b3a3, State: Initialized, Role: FOLLOWER
I20260927 18:38:23.054911 4710 tablet_replica.cc:439] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } }
I20260927 18:38:23.055088 4704 ts_tablet_manager.cc:1428] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:38:23.055217 4710 tablet_replica.cc:442] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:23.055716 4705 ts_tablet_manager.cc:1428] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:38:23.055555 4709 tablet_replica.cc:439] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } }
I20260927 18:38:23.055953 4709 tablet_replica.cc:442] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:23.055117 4703 consensus_queue.cc:260] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.056394 4711 tablet_replica.cc:439] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } }
I20260927 18:38:23.056689 4711 tablet_replica.cc:442] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:23.057801 4703 ts_tablet_manager.cc:1428] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 18:38:23.062088 4709 raft_consensus.cc:482] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:23.062332 4709 raft_consensus.cc:504] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.064194 4709 leader_election.cc:288] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 353d30df674c43f09eb1d1a19f759061 (127.2.95.66:45243), 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655)
I20260927 18:38:23.073854 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.064539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:52549 (local address 127.2.95.66:45243)
0927 18:38:23.064775 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.064783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.065482 (+ 699us) server_negotiation.cc:407] Connection header received
0927 18:38:23.065803 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.065810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.065895 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.065988 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:23.066760 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.067887 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.071116 (+ 3229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.072544 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.072706 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.072822 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.073354 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.073506 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:38:23.073875 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.065223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:60403 (local address 127.2.95.67:40655)
0927 18:38:23.065423 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.065428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.065768 (+ 340us) server_negotiation.cc:407] Connection header received
0927 18:38:23.065906 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.065911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.066001 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.066087 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:23.066760 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.067949 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.071106 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.072549 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.072705 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.072817 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.073353 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.073506 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:38:23.074214 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.064479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:45243 (local address 127.2.95.65:52549)
0927 18:38:23.065351 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:23.065374 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:23.065608 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:23.066015 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:23.066018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:23.066039 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:23.066577 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.066591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.068061 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.068066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.070900 (+ 2834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.070921 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.072848 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.072853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.073116 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.073201 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 18:38:23.073303 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 18:38:23.073524 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":395,"thread_start_us":838,"threads_started":1}
I20260927 18:38:23.074280 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.064536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:40655 (local address 127.2.95.65:60403)
0927 18:38:23.065658 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:23.065700 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:23.065811 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:23.066108 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:23.066112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:23.066188 (+ 76us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:23.066570 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.066609 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.068082 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.068086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.070900 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.070921 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.072830 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.072837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.073104 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.073203 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 18:38:23.073303 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 18:38:23.073524 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":176,"spinlock_wait_cycles":41728,"thread_start_us":473,"threads_started":1}
I20260927 18:38:23.075073 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1db65b0021a464081f47ae0b8d4f15b" candidate_uuid: "5b8fc5f189944a62ba5a60aba519ed25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" is_pre_election: true
I20260927 18:38:23.075402 4634 raft_consensus.cc:2454] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b8fc5f189944a62ba5a60aba519ed25 in term 0.
I20260927 18:38:23.075317 4494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1db65b0021a464081f47ae0b8d4f15b" candidate_uuid: "5b8fc5f189944a62ba5a60aba519ed25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353d30df674c43f09eb1d1a19f759061" is_pre_election: true
I20260927 18:38:23.075825 4494 raft_consensus.cc:2454] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b8fc5f189944a62ba5a60aba519ed25 in term 0.
I20260927 18:38:23.075943 4286 leader_election.cc:302] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dd4ea9d66c843fcb607f0a73ba5b3a3, 5b8fc5f189944a62ba5a60aba519ed25; no voters:
I20260927 18:38:23.076390 4709 raft_consensus.cc:2781] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:23.076506 4709 raft_consensus.cc:482] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:23.076617 4709 raft_consensus.cc:3037] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:23.079131 4709 raft_consensus.cc:504] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.079802 4709 leader_election.cc:288] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [CANDIDATE]: Term 1 election: Requested vote from peers 353d30df674c43f09eb1d1a19f759061 (127.2.95.66:45243), 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655)
I20260927 18:38:23.080278 4494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1db65b0021a464081f47ae0b8d4f15b" candidate_uuid: "5b8fc5f189944a62ba5a60aba519ed25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353d30df674c43f09eb1d1a19f759061"
I20260927 18:38:23.080297 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1db65b0021a464081f47ae0b8d4f15b" candidate_uuid: "5b8fc5f189944a62ba5a60aba519ed25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3"
I20260927 18:38:23.080467 4494 raft_consensus.cc:3037] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:23.080495 4634 raft_consensus.cc:3037] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:23.082837 4634 raft_consensus.cc:2454] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b8fc5f189944a62ba5a60aba519ed25 in term 1.
I20260927 18:38:23.082988 4494 raft_consensus.cc:2454] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b8fc5f189944a62ba5a60aba519ed25 in term 1.
I20260927 18:38:23.083287 4286 leader_election.cc:302] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dd4ea9d66c843fcb607f0a73ba5b3a3, 5b8fc5f189944a62ba5a60aba519ed25; no voters:
I20260927 18:38:23.083593 4709 raft_consensus.cc:2781] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:23.084460 4709 raft_consensus.cc:686] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 LEADER]: Becoming Leader. State: Replica: 5b8fc5f189944a62ba5a60aba519ed25, State: Running, Role: LEADER
I20260927 18:38:23.085009 4709 consensus_queue.cc:237] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.088912 4727 tablet_replica.cc:439] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: TxnStatusTablet state changed. Reason: New leader 5b8fc5f189944a62ba5a60aba519ed25. Latest consensus state: current_term: 1 leader_uuid: "5b8fc5f189944a62ba5a60aba519ed25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } }
I20260927 18:38:23.089287 4727 tablet_replica.cc:442] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:38:23.090050 4729 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:38:23.090433 4729 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:38:23.091285 4195 catalog_manager.cc:5495] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65). New cstate: current_term: 1 leader_uuid: "5b8fc5f189944a62ba5a60aba519ed25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } health_report { overall_health: UNKNOWN } } }
I20260927 18:38:23.126399 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.119044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34915 (local address 127.0.0.1:49592)
0927 18:38:23.119312 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:23.119334 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:23.119484 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:23.120150 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:23.120154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:23.120166 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:38:23.120351 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.120360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.121961 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.121966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.123104 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.123118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.124136 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.124141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.124320 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.124393 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:38:23.125838 (+ 1445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:38:23.125864 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 18:38:23.125938 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 18:38:23.126097 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 18:38:23.126471 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.119051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.2.95.65:34915)
0927 18:38:23.119283 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.119292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.119407 (+ 115us) server_negotiation.cc:407] Connection header received
0927 18:38:23.119739 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.119750 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.119902 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.120075 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:23.120484 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.121800 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.123250 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.123964 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.124107 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.124607 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:23.125656 (+ 1049us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:23.125885 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.125985 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.126096 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 18:38:23.136327 4494 raft_consensus.cc:1261] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Refusing update from remote peer 5b8fc5f189944a62ba5a60aba519ed25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:23.136499 4634 raft_consensus.cc:1261] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Refusing update from remote peer 5b8fc5f189944a62ba5a60aba519ed25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:23.137493 4727 consensus_queue.cc:1035] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:23.137898 4709 consensus_queue.cc:1035] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:23.149716 4710 tablet_replica.cc:439] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061: TxnStatusTablet state changed. Reason: New leader 5b8fc5f189944a62ba5a60aba519ed25. Latest consensus state: current_term: 1 leader_uuid: "5b8fc5f189944a62ba5a60aba519ed25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } }
I20260927 18:38:23.150163 4710 tablet_replica.cc:442] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:23.154487 4709 tablet_replica.cc:439] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b8fc5f189944a62ba5a60aba519ed25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } }
I20260927 18:38:23.154850 4709 tablet_replica.cc:442] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:38:23.158390 4732 mvcc.cc:205] Tried to move back new op lower bound from 7334028505634467840 to 7334028505441001472. Current Snapshot: MvccSnapshot[applied={T|T < 7334028505634467840 or (T in {7334028505634467840})}]
I20260927 18:38:23.164156 4710 tablet_replica.cc:439] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b8fc5f189944a62ba5a60aba519ed25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } }
I20260927 18:38:23.164532 4710 tablet_replica.cc:442] T d1db65b0021a464081f47ae0b8d4f15b P 353d30df674c43f09eb1d1a19f759061: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:23.171649 4711 tablet_replica.cc:439] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: TxnStatusTablet state changed. Reason: New leader 5b8fc5f189944a62ba5a60aba519ed25. Latest consensus state: current_term: 1 leader_uuid: "5b8fc5f189944a62ba5a60aba519ed25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } }
I20260927 18:38:23.171967 4711 tablet_replica.cc:442] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:23.174064 4727 tablet_replica.cc:439] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b8fc5f189944a62ba5a60aba519ed25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } }
I20260927 18:38:23.174391 4727 tablet_replica.cc:442] T d1db65b0021a464081f47ae0b8d4f15b P 5b8fc5f189944a62ba5a60aba519ed25: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:38:23.185791 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.170264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.2.95.66:45243)
0927 18:38:23.170479 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.170487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.172140 (+ 1653us) server_negotiation.cc:407] Connection header received
0927 18:38:23.172515 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.172521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.172682 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.172825 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:23.173621 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.174750 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.182091 (+ 7341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.182823 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.182946 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.183711 (+ 765us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:23.184729 (+ 1018us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:23.184898 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.185185 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.185304 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:38:23.190879 4711 tablet_replica.cc:439] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b8fc5f189944a62ba5a60aba519ed25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } } }
I20260927 18:38:23.191196 4711 tablet_replica.cc:442] T d1db65b0021a464081f47ae0b8d4f15b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:38:23.208489 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.201595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:32823 (local address 127.2.95.65:34915)
0927 18:38:23.201865 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.201873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.202350 (+ 477us) server_negotiation.cc:407] Connection header received
0927 18:38:23.202538 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.202545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.202651 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.202748 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:23.203299 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.204505 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.206166 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.206882 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.207013 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.207623 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:23.207810 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:23.207902 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.208057 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.208154 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260927 18:38:23.208498 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.201452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34915 (local address 127.2.95.66:32823)
0927 18:38:23.202237 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:23.202256 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:23.202404 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:23.202783 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:23.202787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:23.202797 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:38:23.203129 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.203140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.204660 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.204665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.206008 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.206026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.207105 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.207111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.207297 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.207382 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:38:23.207931 (+ 549us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:38:23.207943 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 18:38:23.208011 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 18:38:23.208155 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":417,"threads_started":1}
I20260927 18:38:23.221122 4354 raft_consensus.cc:1261] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Refusing update from remote peer 353d30df674c43f09eb1d1a19f759061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:23.222021 4634 raft_consensus.cc:1261] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Refusing update from remote peer 353d30df674c43f09eb1d1a19f759061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:23.222146 4714 consensus_queue.cc:1035] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:23.222978 4714 consensus_queue.cc:1035] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:23.264997 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.257560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:60591 (local address 127.2.95.66:45243)
0927 18:38:23.257852 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.257860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.258576 (+ 716us) server_negotiation.cc:407] Connection header received
0927 18:38:23.258807 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.258814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.258910 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.259000 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:23.259766 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.260667 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.262929 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.263571 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.263682 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.264272 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:23.264407 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:23.264479 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.264652 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.264735 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:38:23.265216 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.257607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:45243 (local address 127.2.95.65:60591)
0927 18:38:23.258474 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:23.258492 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:23.258643 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:23.259176 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:23.259182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:23.259194 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:38:23.259563 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.259575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.260800 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.260806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.262775 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.262795 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.263761 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.263768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.263970 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.264061 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:38:23.264499 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:38:23.264511 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 18:38:23.264595 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 18:38:23.264771 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":444,"threads_started":1}
I20260927 18:38:23.312860 4195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:23.313258 4195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:23.318300 4195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3: Sending DeleteTablet for 3 replicas of tablet e953c64d22a64992ae1a700311708c3b
I20260927 18:38:23.319859 4334 tablet_service.cc:1514] Processing DeleteTablet for tablet e953c64d22a64992ae1a700311708c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:23 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:49580
I20260927 18:38:23.319850 4614 tablet_service.cc:1514] Processing DeleteTablet for tablet e953c64d22a64992ae1a700311708c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:23 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:37010
I20260927 18:38:23.322407 4474 tablet_service.cc:1514] Processing DeleteTablet for tablet e953c64d22a64992ae1a700311708c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:23 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:57568
I20260927 18:38:23.330389 4757 tablet_replica.cc:330] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061: stopping tablet replica
I20260927 18:38:23.330626 4195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 18:38:23.331053 4757 raft_consensus.cc:2229] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:38:23.331285 4755 tablet_replica.cc:330] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25: stopping tablet replica
I20260927 18:38:23.331822 4755 raft_consensus.cc:2229] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:23.331948 4757 raft_consensus.cc:2258] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 18:38:23.331878 4195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:23.332191 4755 raft_consensus.cc:2258] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:23.333053 4756 tablet_replica.cc:330] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: stopping tablet replica
I20260927 18:38:23.333489 4756 raft_consensus.cc:2229] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:23.333936 4756 raft_consensus.cc:2258] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:23.334381 4757 ts_tablet_manager.cc:1905] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:23.336242 4755 ts_tablet_manager.cc:1905] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:23.338323 4756 ts_tablet_manager.cc:1905] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:23.342450 4334 tablet_service.cc:1467] Processing CreateTablet for tablet 37dd0def47a744c0a4372fd4df400a0e (DEFAULT_TABLE table=test-table [id=9176f2438b754ec2bb69f29c8a346ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:23.342535 4614 tablet_service.cc:1467] Processing CreateTablet for tablet 37dd0def47a744c0a4372fd4df400a0e (DEFAULT_TABLE table=test-table [id=9176f2438b754ec2bb69f29c8a346ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:23.343312 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dd0def47a744c0a4372fd4df400a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:23.343273 4474 tablet_service.cc:1467] Processing CreateTablet for tablet 37dd0def47a744c0a4372fd4df400a0e (DEFAULT_TABLE table=test-table [id=9176f2438b754ec2bb69f29c8a346ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:23.343705 4474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dd0def47a744c0a4372fd4df400a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:23.349356 4334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dd0def47a744c0a4372fd4df400a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:23.352909 4757 ts_tablet_manager.cc:1918] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:38:23.353116 4757 log.cc:1196] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/e953c64d22a64992ae1a700311708c3b
I20260927 18:38:23.353701 4757 ts_tablet_manager.cc:1939] T e953c64d22a64992ae1a700311708c3b P 353d30df674c43f09eb1d1a19f759061: Deleting consensus metadata
I20260927 18:38:23.355861 4181 catalog_manager.cc:4841] TS 353d30df674c43f09eb1d1a19f759061 (127.2.95.66:45243): tablet e953c64d22a64992ae1a700311708c3b (table test-table [id=bd194e8c4187478c9ebc4deffe129e05]) successfully deleted
I20260927 18:38:23.356153 4703 tablet_bootstrap.cc:492] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Bootstrap starting.
I20260927 18:38:23.358076 4703 tablet_bootstrap.cc:654] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:23.360224 4704 tablet_bootstrap.cc:492] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25: Bootstrap starting.
I20260927 18:38:23.362422 4704 tablet_bootstrap.cc:654] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:23.366027 4705 tablet_bootstrap.cc:492] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061: Bootstrap starting.
I20260927 18:38:23.367996 4705 tablet_bootstrap.cc:654] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:23.371707 4703 tablet_bootstrap.cc:492] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3: No bootstrap required, opened a new log
I20260927 18:38:23.371922 4703 ts_tablet_manager.cc:1397] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20260927 18:38:23.372861 4703 raft_consensus.cc:348] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.373152 4703 raft_consensus.cc:374] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:23.373234 4703 raft_consensus.cc:729] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd4ea9d66c843fcb607f0a73ba5b3a3, State: Initialized, Role: FOLLOWER
I20260927 18:38:23.373507 4703 consensus_queue.cc:260] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.374161 4703 ts_tablet_manager.cc:1428] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:38:23.384577 4704 tablet_bootstrap.cc:492] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25: No bootstrap required, opened a new log
I20260927 18:38:23.384938 4704 ts_tablet_manager.cc:1397] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.009s
I20260927 18:38:23.385442 4705 tablet_bootstrap.cc:492] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061: No bootstrap required, opened a new log
I20260927 18:38:23.385658 4705 ts_tablet_manager.cc:1397] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.006s
I20260927 18:38:23.386198 4704 raft_consensus.cc:348] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.386623 4704 raft_consensus.cc:374] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:23.386760 4704 raft_consensus.cc:729] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8fc5f189944a62ba5a60aba519ed25, State: Initialized, Role: FOLLOWER
I20260927 18:38:23.386751 4705 raft_consensus.cc:348] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.387070 4705 raft_consensus.cc:374] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:23.387157 4705 raft_consensus.cc:729] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353d30df674c43f09eb1d1a19f759061, State: Initialized, Role: FOLLOWER
I20260927 18:38:23.387444 4704 consensus_queue.cc:260] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.387512 4705 consensus_queue.cc:260] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.388305 4705 ts_tablet_manager.cc:1428] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:38:23.392261 4704 ts_tablet_manager.cc:1428] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260927 18:38:23.392741 4755 ts_tablet_manager.cc:1918] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:38:23.392951 4755 log.cc:1196] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/e953c64d22a64992ae1a700311708c3b
I20260927 18:38:23.393514 4755 ts_tablet_manager.cc:1939] T e953c64d22a64992ae1a700311708c3b P 5b8fc5f189944a62ba5a60aba519ed25: Deleting consensus metadata
I20260927 18:38:23.395551 4181 catalog_manager.cc:4841] TS 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65:34915): tablet e953c64d22a64992ae1a700311708c3b (table test-table [id=bd194e8c4187478c9ebc4deffe129e05]) successfully deleted
I20260927 18:38:23.397070 4756 ts_tablet_manager.cc:1918] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:38:23.397372 4756 log.cc:1196] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/e953c64d22a64992ae1a700311708c3b
I20260927 18:38:23.398026 4756 ts_tablet_manager.cc:1939] T e953c64d22a64992ae1a700311708c3b P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Deleting consensus metadata
I20260927 18:38:23.400187 4183 catalog_manager.cc:4841] TS 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655): tablet e953c64d22a64992ae1a700311708c3b (table test-table [id=bd194e8c4187478c9ebc4deffe129e05]) successfully deleted
I20260927 18:38:23.496470 4711 raft_consensus.cc:482] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:23.496881 4711 raft_consensus.cc:504] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.499383 4711 leader_election.cc:288] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65:34915), 353d30df674c43f09eb1d1a19f759061 (127.2.95.66:45243)
I20260927 18:38:23.509241 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.499878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:37693 (local address 127.2.95.65:34915)
0927 18:38:23.500129 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.500137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.500891 (+ 754us) server_negotiation.cc:407] Connection header received
0927 18:38:23.501390 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.501396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.501488 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.501566 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:23.502532 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.503735 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.506429 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.507882 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.507995 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.508121 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.508737 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.508894 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:38:23.510473 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.499690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34915 (local address 127.2.95.67:37693)
0927 18:38:23.500756 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:23.500780 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:23.501045 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:23.501843 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:23.501848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:23.501878 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:23.502244 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.502261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.503948 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.503954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.506228 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.506244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.508313 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.508320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.508518 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.508556 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:38:23.509934 (+ 1378us) client_negotiation.cc:240] Negotiation successful
0927 18:38:23.510094 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":719,"threads_started":1}
I20260927 18:38:23.513072 4354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dd0def47a744c0a4372fd4df400a0e" candidate_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8fc5f189944a62ba5a60aba519ed25" is_pre_election: true
I20260927 18:38:23.513415 4354 raft_consensus.cc:2454] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dd4ea9d66c843fcb607f0a73ba5b3a3 in term 0.
I20260927 18:38:23.514196 4564 leader_election.cc:302] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dd4ea9d66c843fcb607f0a73ba5b3a3, 5b8fc5f189944a62ba5a60aba519ed25; no voters:
I20260927 18:38:23.514801 4711 raft_consensus.cc:2781] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:23.514945 4711 raft_consensus.cc:482] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:23.515045 4711 raft_consensus.cc:3037] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:23.518806 4711 raft_consensus.cc:504] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.519712 4711 leader_election.cc:288] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [CANDIDATE]: Term 1 election: Requested vote from peers 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65:34915), 353d30df674c43f09eb1d1a19f759061 (127.2.95.66:45243)
I20260927 18:38:23.520656 4354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dd0def47a744c0a4372fd4df400a0e" candidate_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8fc5f189944a62ba5a60aba519ed25"
I20260927 18:38:23.520936 4354 raft_consensus.cc:3037] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:23.523540 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.501525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:59165 (local address 127.2.95.66:45243)
0927 18:38:23.501757 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.501763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.511086 (+ 9323us) server_negotiation.cc:407] Connection header received
0927 18:38:23.511391 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.511398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.511492 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.511558 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:23.512190 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.513397 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.519673 (+ 6276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.521145 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.521272 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.521404 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.523059 (+ 1655us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.523237 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:38:23.524499 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.500922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:45243 (local address 127.2.95.67:59165)
0927 18:38:23.510968 (+ 10046us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:23.510990 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:23.511215 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:23.511768 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:23.511772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:23.511799 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:23.511996 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.512007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.513819 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.513825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.519369 (+ 5544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.519415 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.521662 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.521708 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.522845 (+ 1137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.522887 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 18:38:23.524008 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0927 18:38:23.524201 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9931,"thread_start_us":11318,"threads_started":1}
I20260927 18:38:23.524785 4354 raft_consensus.cc:2454] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dd4ea9d66c843fcb607f0a73ba5b3a3 in term 1.
I20260927 18:38:23.525498 4564 leader_election.cc:302] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dd4ea9d66c843fcb607f0a73ba5b3a3, 5b8fc5f189944a62ba5a60aba519ed25; no voters:
I20260927 18:38:23.526450 4711 raft_consensus.cc:2781] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:23.526376 4494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dd0def47a744c0a4372fd4df400a0e" candidate_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353d30df674c43f09eb1d1a19f759061" is_pre_election: true
I20260927 18:38:23.526672 4494 raft_consensus.cc:2454] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dd4ea9d66c843fcb607f0a73ba5b3a3 in term 0.
I20260927 18:38:23.526679 4493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dd0def47a744c0a4372fd4df400a0e" candidate_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353d30df674c43f09eb1d1a19f759061"
I20260927 18:38:23.526961 4493 raft_consensus.cc:3037] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:23.530463 4493 raft_consensus.cc:2454] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dd4ea9d66c843fcb607f0a73ba5b3a3 in term 1.
I20260927 18:38:23.531423 4711 raft_consensus.cc:686] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2dd4ea9d66c843fcb607f0a73ba5b3a3, State: Running, Role: LEADER
I20260927 18:38:23.531982 4711 consensus_queue.cc:237] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:23.537168 4196 catalog_manager.cc:5495] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67). New cstate: current_term: 1 leader_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } health_report { overall_health: HEALTHY } } }
I20260927 18:38:23.609052 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.590299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.2.95.67:40655)
0927 18:38:23.600386 (+ 10087us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.600395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.600430 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:38:23.600573 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.600579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.600754 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.601972 (+ 1218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:23.602156 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.603444 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.605513 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.606245 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.606386 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.606978 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:23.607973 (+ 995us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:23.608132 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.608276 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.608393 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9870,"thread_start_us":10686,"threads_started":1}
I20260927 18:38:23.631932 4768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.624800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34915 (local address 127.2.95.67:47179)
0927 18:38:23.625580 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:23.625601 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:23.625839 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:23.626251 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:23.626256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:23.626279 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:38:23.626640 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.626652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.628170 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.628175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.629747 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.629764 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.630742 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.630748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.630872 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.630936 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:38:23.631398 (+ 462us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:38:23.631406 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:38:23.631464 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 18:38:23.631584 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":378,"threads_started":1}
I20260927 18:38:23.631937 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.624812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:47179 (local address 127.2.95.65:34915)
0927 18:38:23.625009 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.625017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.625752 (+ 735us) server_negotiation.cc:407] Connection header received
0927 18:38:23.625998 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.626004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.626117 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.626220 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:23.626849 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.627986 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.629890 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.630541 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.630686 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.631135 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:23.631306 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:23.631408 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.631512 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.631615 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:38:23.644474 4354 raft_consensus.cc:1261] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Refusing update from remote peer 2dd4ea9d66c843fcb607f0a73ba5b3a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:23.644469 4493 raft_consensus.cc:1261] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Refusing update from remote peer 2dd4ea9d66c843fcb607f0a73ba5b3a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:23.645339 4763 consensus_queue.cc:1035] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:23.646832 4711 consensus_queue.cc:1035] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:23.669415 4733 mvcc.cc:205] Tried to move back new op lower bound from 7334028507720798208 to 7334028507273265152. Current Snapshot: MvccSnapshot[applied={T|T < 7334028507720798208}]
I20260927 18:38:23.687783 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.682229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:40655 (local address 127.2.95.65:42635)
0927 18:38:23.682474 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:23.682494 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:23.682629 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:23.682931 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:23.682936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:23.682948 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:38:23.683158 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.683180 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.684404 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.684410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.685802 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:23.685816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.686521 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.686527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:23.686718 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.686773 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:38:23.687278 (+ 505us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:38:23.687293 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 18:38:23.687372 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:38:23.687514 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 18:38:23.688369 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:23.682286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:42635 (local address 127.2.95.67:40655)
0927 18:38:23.682457 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:23.682464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:23.682563 (+ 99us) server_negotiation.cc:407] Connection header received
0927 18:38:23.682752 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:23.682758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:23.682833 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:23.682905 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:23.683329 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.684238 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.685932 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:23.686380 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:23.686491 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:23.686978 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:23.687152 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:23.687248 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:23.688015 (+ 767us) server_negotiation.cc:299] Negotiation successful
0927 18:38:23.688113 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 18:38:23.689559 4734 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 18:38:25.367620 4196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:25.367971 4196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:25.373711 4196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3: Sending DeleteTablet for 3 replicas of tablet 37dd0def47a744c0a4372fd4df400a0e
I20260927 18:38:25.374835 4614 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dd0def47a744c0a4372fd4df400a0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:25 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:37010
I20260927 18:38:25.374890 4334 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dd0def47a744c0a4372fd4df400a0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:25 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:49580
I20260927 18:38:25.375275 4474 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dd0def47a744c0a4372fd4df400a0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:25 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:57568
I20260927 18:38:25.376956 4798 tablet_replica.cc:330] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3: stopping tablet replica
I20260927 18:38:25.377230 4798 raft_consensus.cc:2229] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:38:25.378055 4798 raft_consensus.cc:2258] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:25.379434 4798 ts_tablet_manager.cc:1905] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:25.387315 4800 tablet_replica.cc:330] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061: stopping tablet replica
I20260927 18:38:25.387557 4800 raft_consensus.cc:2229] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:25.387789 4800 raft_consensus.cc:2258] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:25.387872 4798 ts_tablet_manager.cc:1918] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:38:25.388185 4798 log.cc:1196] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/37dd0def47a744c0a4372fd4df400a0e
I20260927 18:38:25.388718 4798 ts_tablet_manager.cc:1939] T 37dd0def47a744c0a4372fd4df400a0e P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Deleting consensus metadata
I20260927 18:38:25.388926 4800 ts_tablet_manager.cc:1905] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:25.390123 4799 tablet_replica.cc:330] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25: stopping tablet replica
I20260927 18:38:25.390352 4799 raft_consensus.cc:2229] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:25.390594 4799 raft_consensus.cc:2258] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:25.390995 4183 catalog_manager.cc:4841] TS 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655): tablet 37dd0def47a744c0a4372fd4df400a0e (table test-table [id=9176f2438b754ec2bb69f29c8a346ce0]) successfully deleted
I20260927 18:38:25.391866 4799 ts_tablet_manager.cc:1905] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:25.395730 4800 ts_tablet_manager.cc:1918] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:38:25.395921 4800 log.cc:1196] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/37dd0def47a744c0a4372fd4df400a0e
I20260927 18:38:25.396399 4800 ts_tablet_manager.cc:1939] T 37dd0def47a744c0a4372fd4df400a0e P 353d30df674c43f09eb1d1a19f759061: Deleting consensus metadata
I20260927 18:38:25.397761 4181 catalog_manager.cc:4841] TS 353d30df674c43f09eb1d1a19f759061 (127.2.95.66:45243): tablet 37dd0def47a744c0a4372fd4df400a0e (table test-table [id=9176f2438b754ec2bb69f29c8a346ce0]) successfully deleted
I20260927 18:38:25.398530 4799 ts_tablet_manager.cc:1918] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:38:25.398644 4799 log.cc:1196] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/37dd0def47a744c0a4372fd4df400a0e
I20260927 18:38:25.398994 4799 ts_tablet_manager.cc:1939] T 37dd0def47a744c0a4372fd4df400a0e P 5b8fc5f189944a62ba5a60aba519ed25: Deleting consensus metadata
I20260927 18:38:25.402820 4181 catalog_manager.cc:4841] TS 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65:34915): tablet 37dd0def47a744c0a4372fd4df400a0e (table test-table [id=9176f2438b754ec2bb69f29c8a346ce0]) successfully deleted
Sep 27 18:38:25 dist-test-slave-9jll krb5kdc[4152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:38:25 dist-test-slave-9jll krb5kdc[4152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:25.438210 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:25.420985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.2.95.126:45801)
0927 18:38:25.421633 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:25.421638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:25.421719 (+ 81us) server_negotiation.cc:407] Connection header received
0927 18:38:25.421794 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:25.421799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:25.421883 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:25.422015 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:25.423536 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.424451 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.425946 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.426411 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.426524 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:25.431353 (+ 4829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:25.431360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:25.431364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:25.431420 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:25.435098 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:25.436065 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:25.436070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:25.436072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:25.436242 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:25.436603 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:25.436606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:25.436608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:25.436961 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:25.437150 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:25.437592 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 18:38:25.437883 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":371,"threads_started":1}
I20260927 18:38:25.443096 4196 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:38:25.443878 4196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:25.452320 4334 tablet_service.cc:1467] Processing CreateTablet for tablet 155ae29e1e3e4b5ba40de45a11e3988c (DEFAULT_TABLE table=test-table [id=3e19c9ec1ce147f7bb84d2056888d3d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:25.452473 4474 tablet_service.cc:1467] Processing CreateTablet for tablet 155ae29e1e3e4b5ba40de45a11e3988c (DEFAULT_TABLE table=test-table [id=3e19c9ec1ce147f7bb84d2056888d3d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:25.452613 4614 tablet_service.cc:1467] Processing CreateTablet for tablet 155ae29e1e3e4b5ba40de45a11e3988c (DEFAULT_TABLE table=test-table [id=3e19c9ec1ce147f7bb84d2056888d3d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:25.452967 4334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 155ae29e1e3e4b5ba40de45a11e3988c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:25.452982 4474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 155ae29e1e3e4b5ba40de45a11e3988c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:25.453006 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 155ae29e1e3e4b5ba40de45a11e3988c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:25.461360 4810 tablet_bootstrap.cc:492] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061: Bootstrap starting.
I20260927 18:38:25.462440 4809 tablet_bootstrap.cc:492] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25: Bootstrap starting.
I20260927 18:38:25.464349 4810 tablet_bootstrap.cc:654] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:25.464754 4809 tablet_bootstrap.cc:654] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:25.466390 4811 tablet_bootstrap.cc:492] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Bootstrap starting.
I20260927 18:38:25.467621 4809 tablet_bootstrap.cc:492] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25: No bootstrap required, opened a new log
I20260927 18:38:25.467609 4810 tablet_bootstrap.cc:492] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061: No bootstrap required, opened a new log
I20260927 18:38:25.467810 4810 ts_tablet_manager.cc:1397] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 18:38:25.467878 4809 ts_tablet_manager.cc:1397] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 18:38:25.468729 4809 raft_consensus.cc:348] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.468736 4810 raft_consensus.cc:348] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.469017 4809 raft_consensus.cc:374] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:25.469017 4810 raft_consensus.cc:374] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:25.469120 4810 raft_consensus.cc:729] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353d30df674c43f09eb1d1a19f759061, State: Initialized, Role: FOLLOWER
I20260927 18:38:25.469115 4809 raft_consensus.cc:729] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8fc5f189944a62ba5a60aba519ed25, State: Initialized, Role: FOLLOWER
I20260927 18:38:25.469419 4811 tablet_bootstrap.cc:654] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:25.469419 4810 consensus_queue.cc:260] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.469420 4809 consensus_queue.cc:260] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.470111 4809 ts_tablet_manager.cc:1428] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:38:25.471422 4810 ts_tablet_manager.cc:1428] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 18:38:25.472460 4811 tablet_bootstrap.cc:492] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3: No bootstrap required, opened a new log
I20260927 18:38:25.472734 4811 ts_tablet_manager.cc:1397] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260927 18:38:25.473743 4811 raft_consensus.cc:348] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.473964 4811 raft_consensus.cc:374] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:25.474061 4811 raft_consensus.cc:729] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd4ea9d66c843fcb607f0a73ba5b3a3, State: Initialized, Role: FOLLOWER
I20260927 18:38:25.474306 4811 consensus_queue.cc:260] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.474897 4811 ts_tablet_manager.cc:1428] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 18:38:25.633388 4753 raft_consensus.cc:482] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:25.633769 4753 raft_consensus.cc:504] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.634644 4753 leader_election.cc:288] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 353d30df674c43f09eb1d1a19f759061 (127.2.95.66:45243), 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655)
I20260927 18:38:25.635385 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "155ae29e1e3e4b5ba40de45a11e3988c" candidate_uuid: "5b8fc5f189944a62ba5a60aba519ed25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" is_pre_election: true
I20260927 18:38:25.635388 4493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "155ae29e1e3e4b5ba40de45a11e3988c" candidate_uuid: "5b8fc5f189944a62ba5a60aba519ed25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353d30df674c43f09eb1d1a19f759061" is_pre_election: true
I20260927 18:38:25.635676 4493 raft_consensus.cc:2454] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b8fc5f189944a62ba5a60aba519ed25 in term 0.
I20260927 18:38:25.635677 4634 raft_consensus.cc:2454] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b8fc5f189944a62ba5a60aba519ed25 in term 0.
I20260927 18:38:25.636281 4284 leader_election.cc:302] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2dd4ea9d66c843fcb607f0a73ba5b3a3, 353d30df674c43f09eb1d1a19f759061, 5b8fc5f189944a62ba5a60aba519ed25; no voters:
I20260927 18:38:25.636646 4753 raft_consensus.cc:2781] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:25.636824 4753 raft_consensus.cc:482] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:25.636981 4753 raft_consensus.cc:3037] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:25.639227 4753 raft_consensus.cc:504] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.639912 4753 leader_election.cc:288] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [CANDIDATE]: Term 1 election: Requested vote from peers 353d30df674c43f09eb1d1a19f759061 (127.2.95.66:45243), 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655)
I20260927 18:38:25.640585 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "155ae29e1e3e4b5ba40de45a11e3988c" candidate_uuid: "5b8fc5f189944a62ba5a60aba519ed25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3"
I20260927 18:38:25.640652 4493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "155ae29e1e3e4b5ba40de45a11e3988c" candidate_uuid: "5b8fc5f189944a62ba5a60aba519ed25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353d30df674c43f09eb1d1a19f759061"
I20260927 18:38:25.640823 4634 raft_consensus.cc:3037] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:25.640928 4493 raft_consensus.cc:3037] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:25.643183 4634 raft_consensus.cc:2454] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b8fc5f189944a62ba5a60aba519ed25 in term 1.
I20260927 18:38:25.643682 4286 leader_election.cc:302] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dd4ea9d66c843fcb607f0a73ba5b3a3, 5b8fc5f189944a62ba5a60aba519ed25; no voters:
I20260927 18:38:25.643972 4753 raft_consensus.cc:2781] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:25.644217 4753 raft_consensus.cc:686] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 LEADER]: Becoming Leader. State: Replica: 5b8fc5f189944a62ba5a60aba519ed25, State: Running, Role: LEADER
I20260927 18:38:25.644225 4493 raft_consensus.cc:2454] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b8fc5f189944a62ba5a60aba519ed25 in term 1.
I20260927 18:38:25.644498 4753 consensus_queue.cc:237] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.648147 4195 catalog_manager.cc:5495] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65). New cstate: current_term: 1 leader_uuid: "5b8fc5f189944a62ba5a60aba519ed25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } health_report { overall_health: UNKNOWN } } }
I20260927 18:38:25.716480 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:25.711374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.2.95.65:34915)
0927 18:38:25.711944 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:25.711949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:25.711969 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:38:25.712032 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:25.712036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:25.712169 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:25.712265 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:25.712615 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.713476 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.714825 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.715257 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.715355 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:25.715811 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:25.715930 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:25.715994 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:25.716128 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 18:38:25.716195 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":322,"threads_started":1}
I20260927 18:38:25.732558 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:25.727688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:53557 (local address 127.2.95.65:34915)
0927 18:38:25.727918 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:25.727925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:25.728168 (+ 243us) server_negotiation.cc:407] Connection header received
0927 18:38:25.728352 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:25.728358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:25.728425 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:25.728494 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:25.728971 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.729652 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.731018 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.731387 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.731493 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:25.731840 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:25.731941 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:25.732012 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:25.732131 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 18:38:25.732196 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:38:25.732564 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:25.727419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34915 (local address 127.2.95.65:53557)
0927 18:38:25.728063 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:25.728079 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:25.728234 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:25.728514 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:25.728519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:25.728529 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:38:25.728824 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:25.728837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.729847 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.729853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:25.730894 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:25.730905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.731535 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.731541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:25.731671 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:25.731705 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:38:25.732024 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:38:25.732035 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:38:25.732107 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:38:25.732238 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":298,"threads_started":1}
I20260927 18:38:25.742048 4493 raft_consensus.cc:1261] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Refusing update from remote peer 5b8fc5f189944a62ba5a60aba519ed25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:25.742143 4634 raft_consensus.cc:1261] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Refusing update from remote peer 5b8fc5f189944a62ba5a60aba519ed25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:25.742666 4792 consensus_queue.cc:1035] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:25.742965 4753 consensus_queue.cc:1035] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:25.798853 4196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:25.799134 4196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:25.802590 4196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3: Sending DeleteTablet for 3 replicas of tablet 155ae29e1e3e4b5ba40de45a11e3988c
I20260927 18:38:25.803562 4614 tablet_service.cc:1514] Processing DeleteTablet for tablet 155ae29e1e3e4b5ba40de45a11e3988c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:25 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:37010
I20260927 18:38:25.803565 4333 tablet_service.cc:1514] Processing DeleteTablet for tablet 155ae29e1e3e4b5ba40de45a11e3988c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:25 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:49580
I20260927 18:38:25.803884 4798 tablet_replica.cc:330] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3: stopping tablet replica
I20260927 18:38:25.803884 4799 tablet_replica.cc:330] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25: stopping tablet replica
I20260927 18:38:25.803784 4474 tablet_service.cc:1514] Processing DeleteTablet for tablet 155ae29e1e3e4b5ba40de45a11e3988c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:25 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:57568
I20260927 18:38:25.804087 4798 raft_consensus.cc:2229] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:25.804100 4799 raft_consensus.cc:2229] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:38:25.804174 4800 tablet_replica.cc:330] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061: stopping tablet replica
I20260927 18:38:25.804279 4798 raft_consensus.cc:2258] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:25.804404 4800 raft_consensus.cc:2229] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:25.804618 4800 raft_consensus.cc:2258] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:25.804667 4799 raft_consensus.cc:2258] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:25.805763 4800 ts_tablet_manager.cc:1905] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:25.807034 4196 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:38:25.807735 4196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:25.808139 4798 ts_tablet_manager.cc:1905] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:25.815189 4800 ts_tablet_manager.cc:1918] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:38:25.815408 4800 log.cc:1196] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/155ae29e1e3e4b5ba40de45a11e3988c
I20260927 18:38:25.815299 4614 tablet_service.cc:1467] Processing CreateTablet for tablet 2878ae17dd3d493f93c37a70a6130b55 (DEFAULT_TABLE table=test-table [id=08b9220ce35c4fbf8f7e55719da1bb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:25.815426 4474 tablet_service.cc:1467] Processing CreateTablet for tablet 2878ae17dd3d493f93c37a70a6130b55 (DEFAULT_TABLE table=test-table [id=08b9220ce35c4fbf8f7e55719da1bb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:25.815773 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2878ae17dd3d493f93c37a70a6130b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:25.815850 4800 ts_tablet_manager.cc:1939] T 155ae29e1e3e4b5ba40de45a11e3988c P 353d30df674c43f09eb1d1a19f759061: Deleting consensus metadata
I20260927 18:38:25.816004 4474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2878ae17dd3d493f93c37a70a6130b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:25.821578 4811 tablet_bootstrap.cc:492] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Bootstrap starting.
I20260927 18:38:25.822140 4181 catalog_manager.cc:4841] TS 353d30df674c43f09eb1d1a19f759061 (127.2.95.66:45243): tablet 155ae29e1e3e4b5ba40de45a11e3988c (table test-table [id=3e19c9ec1ce147f7bb84d2056888d3d5]) successfully deleted
I20260927 18:38:25.823100 4811 tablet_bootstrap.cc:654] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:25.823352 4333 tablet_service.cc:1467] Processing CreateTablet for tablet 2878ae17dd3d493f93c37a70a6130b55 (DEFAULT_TABLE table=test-table [id=08b9220ce35c4fbf8f7e55719da1bb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:25.823704 4799 ts_tablet_manager.cc:1905] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:25.824896 4333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2878ae17dd3d493f93c37a70a6130b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:25.832275 4798 ts_tablet_manager.cc:1918] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:38:25.832482 4798 log.cc:1196] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/155ae29e1e3e4b5ba40de45a11e3988c
I20260927 18:38:25.833005 4798 ts_tablet_manager.cc:1939] T 155ae29e1e3e4b5ba40de45a11e3988c P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Deleting consensus metadata
I20260927 18:38:25.834406 4183 catalog_manager.cc:4841] TS 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655): tablet 155ae29e1e3e4b5ba40de45a11e3988c (table test-table [id=3e19c9ec1ce147f7bb84d2056888d3d5]) successfully deleted
I20260927 18:38:25.836216 4809 tablet_bootstrap.cc:492] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25: Bootstrap starting.
I20260927 18:38:25.836920 4810 tablet_bootstrap.cc:492] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061: Bootstrap starting.
I20260927 18:38:25.838189 4809 tablet_bootstrap.cc:654] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:25.838788 4799 ts_tablet_manager.cc:1918] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:38:25.838948 4799 log.cc:1196] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/155ae29e1e3e4b5ba40de45a11e3988c
I20260927 18:38:25.839440 4799 ts_tablet_manager.cc:1939] T 155ae29e1e3e4b5ba40de45a11e3988c P 5b8fc5f189944a62ba5a60aba519ed25: Deleting consensus metadata
I20260927 18:38:25.839715 4810 tablet_bootstrap.cc:654] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:25.842120 4181 catalog_manager.cc:4841] TS 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65:34915): tablet 155ae29e1e3e4b5ba40de45a11e3988c (table test-table [id=3e19c9ec1ce147f7bb84d2056888d3d5]) successfully deleted
I20260927 18:38:25.843917 4810 tablet_bootstrap.cc:492] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061: No bootstrap required, opened a new log
I20260927 18:38:25.843918 4811 tablet_bootstrap.cc:492] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3: No bootstrap required, opened a new log
I20260927 18:38:25.844156 4810 ts_tablet_manager.cc:1397] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 18:38:25.844156 4811 ts_tablet_manager.cc:1397] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.004s
I20260927 18:38:25.844974 4811 raft_consensus.cc:348] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.844971 4810 raft_consensus.cc:348] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.845255 4811 raft_consensus.cc:374] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:25.845286 4810 raft_consensus.cc:374] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:25.845382 4810 raft_consensus.cc:729] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353d30df674c43f09eb1d1a19f759061, State: Initialized, Role: FOLLOWER
I20260927 18:38:25.845382 4811 raft_consensus.cc:729] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd4ea9d66c843fcb607f0a73ba5b3a3, State: Initialized, Role: FOLLOWER
I20260927 18:38:25.845749 4810 consensus_queue.cc:260] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.845965 4811 consensus_queue.cc:260] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.846546 4811 ts_tablet_manager.cc:1428] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 18:38:25.847137 4810 ts_tablet_manager.cc:1428] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 18:38:25.852106 4809 tablet_bootstrap.cc:492] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25: No bootstrap required, opened a new log
I20260927 18:38:25.852382 4809 ts_tablet_manager.cc:1397] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260927 18:38:25.853127 4809 raft_consensus.cc:348] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.853355 4809 raft_consensus.cc:374] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:25.853456 4809 raft_consensus.cc:729] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8fc5f189944a62ba5a60aba519ed25, State: Initialized, Role: FOLLOWER
I20260927 18:38:25.853739 4809 consensus_queue.cc:260] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.854274 4809 ts_tablet_manager.cc:1428] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 18:38:25.856930 4814 raft_consensus.cc:482] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:25.857120 4814 raft_consensus.cc:504] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.857975 4814 leader_election.cc:288] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65:34915), 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655)
I20260927 18:38:25.859332 4354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2878ae17dd3d493f93c37a70a6130b55" candidate_uuid: "353d30df674c43f09eb1d1a19f759061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8fc5f189944a62ba5a60aba519ed25" is_pre_election: true
I20260927 18:38:25.859710 4354 raft_consensus.cc:2454] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353d30df674c43f09eb1d1a19f759061 in term 0.
I20260927 18:38:25.860189 4424 leader_election.cc:302] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353d30df674c43f09eb1d1a19f759061, 5b8fc5f189944a62ba5a60aba519ed25; no voters:
I20260927 18:38:25.860553 4814 raft_consensus.cc:2781] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:25.860520 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2878ae17dd3d493f93c37a70a6130b55" candidate_uuid: "353d30df674c43f09eb1d1a19f759061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" is_pre_election: true
I20260927 18:38:25.860698 4814 raft_consensus.cc:482] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:25.860723 4634 raft_consensus.cc:2454] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353d30df674c43f09eb1d1a19f759061 in term 0.
I20260927 18:38:25.860785 4814 raft_consensus.cc:3037] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:25.864346 4814 raft_consensus.cc:504] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.866117 4814 leader_election.cc:288] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [CANDIDATE]: Term 1 election: Requested vote from peers 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65:34915), 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655)
I20260927 18:38:25.866942 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2878ae17dd3d493f93c37a70a6130b55" candidate_uuid: "353d30df674c43f09eb1d1a19f759061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3"
I20260927 18:38:25.867218 4634 raft_consensus.cc:3037] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:25.870040 4354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2878ae17dd3d493f93c37a70a6130b55" candidate_uuid: "353d30df674c43f09eb1d1a19f759061" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b8fc5f189944a62ba5a60aba519ed25"
I20260927 18:38:25.870313 4354 raft_consensus.cc:3037] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:25.870553 4634 raft_consensus.cc:2454] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353d30df674c43f09eb1d1a19f759061 in term 1.
I20260927 18:38:25.871259 4426 leader_election.cc:302] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dd4ea9d66c843fcb607f0a73ba5b3a3, 353d30df674c43f09eb1d1a19f759061; no voters:
I20260927 18:38:25.871639 4814 raft_consensus.cc:2781] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:25.872676 4814 raft_consensus.cc:686] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 1 LEADER]: Becoming Leader. State: Replica: 353d30df674c43f09eb1d1a19f759061, State: Running, Role: LEADER
I20260927 18:38:25.873049 4814 consensus_queue.cc:237] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } }
I20260927 18:38:25.873711 4354 raft_consensus.cc:2454] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353d30df674c43f09eb1d1a19f759061 in term 1.
I20260927 18:38:25.876267 4196 catalog_manager.cc:5495] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 reported cstate change: term changed from 0 to 1, leader changed from <none> to 353d30df674c43f09eb1d1a19f759061 (127.2.95.66). New cstate: current_term: 1 leader_uuid: "353d30df674c43f09eb1d1a19f759061" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353d30df674c43f09eb1d1a19f759061" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 45243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 } health_report { overall_health: UNKNOWN } } }
I20260927 18:38:25.902961 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:25.897121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.2.95.66:45243)
0927 18:38:25.897865 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:25.897870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:25.897891 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:38:25.897954 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:25.897958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:25.898025 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:25.898119 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:25.898550 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.899334 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.900637 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:25.901062 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:25.901174 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:25.902139 (+ 965us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:25.902267 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:25.902345 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:25.902487 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 18:38:25.902571 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":402,"threads_started":1}
I20260927 18:38:25.915741 4634 raft_consensus.cc:1261] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Refusing update from remote peer 353d30df674c43f09eb1d1a19f759061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:25.915923 4354 raft_consensus.cc:1261] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Refusing update from remote peer 353d30df674c43f09eb1d1a19f759061: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:25.916700 4829 consensus_queue.cc:1035] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dd4ea9d66c843fcb607f0a73ba5b3a3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:25.917260 4814 consensus_queue.cc:1035] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b8fc5f189944a62ba5a60aba519ed25" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:38:25.950974 4824 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 18:38:27.560992 4195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:27.561298 4195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:27.566262 4195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5041b3cdbd74a928c49fe81bbd645f3: Sending DeleteTablet for 3 replicas of tablet 2878ae17dd3d493f93c37a70a6130b55
I20260927 18:38:27.567291 4614 tablet_service.cc:1514] Processing DeleteTablet for tablet 2878ae17dd3d493f93c37a70a6130b55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:27 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:37010
I20260927 18:38:27.567298 4334 tablet_service.cc:1514] Processing DeleteTablet for tablet 2878ae17dd3d493f93c37a70a6130b55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:27 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:49580
I20260927 18:38:27.567716 4474 tablet_service.cc:1514] Processing DeleteTablet for tablet 2878ae17dd3d493f93c37a70a6130b55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:27 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:57568
I20260927 18:38:27.568239 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 4268
I20260927 18:38:27.579701 4859 tablet_replica.cc:330] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25: stopping tablet replica
I20260927 18:38:27.580174 4859 raft_consensus.cc:2229] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:27.580606 4859 raft_consensus.cc:2258] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:27.581373 4860 tablet_replica.cc:330] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061: stopping tablet replica
I20260927 18:38:27.581866 4860 raft_consensus.cc:2229] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:38:27.582001 4859 ts_tablet_manager.cc:1905] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:27.582465 4860 raft_consensus.cc:2258] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:27.583707 4860 ts_tablet_manager.cc:1905] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:27.585464 4858 tablet_replica.cc:330] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3: stopping tablet replica
I20260927 18:38:27.585728 4858 raft_consensus.cc:2229] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:27.586094 4858 raft_consensus.cc:2258] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:27.587791 4858 ts_tablet_manager.cc:1905] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:27.591877 4859 ts_tablet_manager.cc:1918] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:38:27.592212 4859 log.cc:1196] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/2878ae17dd3d493f93c37a70a6130b55
I20260927 18:38:27.592793 4859 ts_tablet_manager.cc:1939] T 2878ae17dd3d493f93c37a70a6130b55 P 5b8fc5f189944a62ba5a60aba519ed25: Deleting consensus metadata
I20260927 18:38:27.594270 4181 catalog_manager.cc:4841] TS 5b8fc5f189944a62ba5a60aba519ed25 (127.2.95.65:34915): tablet 2878ae17dd3d493f93c37a70a6130b55 (table test-table [id=08b9220ce35c4fbf8f7e55719da1bb0e]) successfully deleted
I20260927 18:38:27.595337 4860 ts_tablet_manager.cc:1918] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:38:27.595511 4860 log.cc:1196] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/2878ae17dd3d493f93c37a70a6130b55
I20260927 18:38:27.595937 4860 ts_tablet_manager.cc:1939] T 2878ae17dd3d493f93c37a70a6130b55 P 353d30df674c43f09eb1d1a19f759061: Deleting consensus metadata
I20260927 18:38:27.597226 4181 catalog_manager.cc:4841] TS 353d30df674c43f09eb1d1a19f759061 (127.2.95.66:45243): tablet 2878ae17dd3d493f93c37a70a6130b55 (table test-table [id=08b9220ce35c4fbf8f7e55719da1bb0e]) successfully deleted
I20260927 18:38:27.598104 4858 ts_tablet_manager.cc:1918] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 18:38:27.598270 4858 log.cc:1196] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/2878ae17dd3d493f93c37a70a6130b55
I20260927 18:38:27.598613 4858 ts_tablet_manager.cc:1939] T 2878ae17dd3d493f93c37a70a6130b55 P 2dd4ea9d66c843fcb607f0a73ba5b3a3: Deleting consensus metadata
I20260927 18:38:27.599649 4183 catalog_manager.cc:4841] TS 2dd4ea9d66c843fcb607f0a73ba5b3a3 (127.2.95.67:40655): tablet 2878ae17dd3d493f93c37a70a6130b55 (table test-table [id=08b9220ce35c4fbf8f7e55719da1bb0e]) successfully deleted
Sep 27 18:38:27 dist-test-slave-9jll krb5kdc[4152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:27.615069 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:27.570569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.2.95.65:34915)
0927 18:38:27.591760 (+ 21191us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:27.591767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:27.592878 (+ 1111us) server_negotiation.cc:407] Connection header received
0927 18:38:27.592958 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:27.592964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:27.593066 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:27.593176 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:27.595311 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:27.596238 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:27.597812 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:27.598319 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:27.598428 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:27.607790 (+ 9362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:27.607813 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:27.607839 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:27.607895 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:27.611123 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:27.612059 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:27.612067 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:27.612069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:27.612281 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:27.612749 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:27.612753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:27.612755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:27.613208 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:27.613481 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:27.614519 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0927 18:38:27.614760 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20858,"spinlock_wait_cycles":641920,"thread_start_us":20917,"threads_started":1}
W20260927 18:38:27.616052 4399 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44620
W20260927 18:38:27.616814 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 4268: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:38:27.617057 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 4268
I20260927 18:38:27.639241 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 4408
Sep 27 18:38:27 dist-test-slave-9jll krb5kdc[4152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:38:27.658006 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:27.640116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.2.95.66:45243)
0927 18:38:27.640770 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:27.640777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:27.640812 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:38:27.640890 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:27.640896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:27.640990 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:27.641142 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:27.642981 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:27.644178 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:27.645769 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:27.646495 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:27.646625 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:27.651581 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:27.651600 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:27.651621 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:27.651667 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:27.654736 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:27.655758 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:27.655768 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:27.655772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:27.655935 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:27.656377 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:27.656386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:27.656390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:27.656830 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:27.657135 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:27.657446 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 18:38:27.657717 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":310,"threads_started":1}
W20260927 18:38:27.658773 4538 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50116
W20260927 18:38:27.659629 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 4408: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:38:27.659835 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 4408
I20260927 18:38:27.681735 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 4548
Sep 27 18:38:27 dist-test-slave-9jll krb5kdc[4152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534305, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:38:27.700377 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:27.682829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53066 (local address 127.2.95.67:40655)
0927 18:38:27.683437 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:27.683442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:27.683463 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:38:27.683525 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:27.683529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:27.683601 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:27.683716 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:27.685372 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:27.686372 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:27.687957 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:27.688410 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:27.688536 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:27.693592 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:27.693617 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:27.693641 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:27.693892 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:27.696982 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:27.698045 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:27.698054 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:27.698057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:27.698258 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:27.698695 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:27.698701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:27.698705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:27.699176 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:27.699445 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:27.699825 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 18:38:27.700045 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":285,"threads_started":1}
W20260927 18:38:27.701201 4679 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53066
W20260927 18:38:27.701982 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 4548: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:38:27.702231 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 4548
I20260927 18:38:27.724642 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 4165
I20260927 18:38:27.726236 4237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:27.843218 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 4165
2026-09-27T18:38:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8924 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:27 dist-test-slave-9jll krb5kdc[4868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:27 dist-test-slave-9jll krb5kdc[4868](info): set up 2 sockets
Sep 27 18:38:27 dist-test-slave-9jll krb5kdc[4868](info): commencing operation
krb5kdc: starting...
W20260927 18:38:28.867448 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:38:29 dist-test-slave-9jll krb5kdc[4868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:38:29.078393 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:46783
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:46783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:29.448535 4881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:29.448928 4881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:29.449055 4881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:29.456835 4881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:29.456997 4881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:29.457078 4881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:29.457154 4881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:29.457222 4881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:29.468253 4881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:46783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:46783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:29.470116 4881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:29.471813 4881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:29.485436 4886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:29.485500 4887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:29.485436 4889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:29.486076 4881 server_base.cc:1034] running on GCE node
I20260927 18:38:29.487032 4881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:29.488709 4881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:29.489913 4881 hybrid_clock.cc:648] HybridClock initialized: now 1790534309489866 us; error 33 us; skew 500 ppm
I20260927 18:38:29.490293 4881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:29 dist-test-slave-9jll krb5kdc[4868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:29.493541 4881 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:38:29.495469 4881 webserver.cc:460] Webserver started at http://127.2.95.126:44721/ using document root <none> and password file <none>
I20260927 18:38:29.496234 4881 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:29.496336 4881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:29.496703 4881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:29.500373 4881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "cd7c197de32e4e9390c0a25e0af05585"
format_stamp: "Formatted at 2026-09-27 18:38:29 on dist-test-slave-9jll"
server_key: "daf22b4891bff5a777c88ff542cb8541"
server_key_iv: "881ea4f6c598647bc528405890a903fc"
server_key_version: "encryptionkey@0"
I20260927 18:38:29.501209 4881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "cd7c197de32e4e9390c0a25e0af05585"
format_stamp: "Formatted at 2026-09-27 18:38:29 on dist-test-slave-9jll"
server_key: "daf22b4891bff5a777c88ff542cb8541"
server_key_iv: "881ea4f6c598647bc528405890a903fc"
server_key_version: "encryptionkey@0"
I20260927 18:38:29.507155 4881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 18:38:29.511067 4896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:29.512616 4881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:38:29.512816 4881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "cd7c197de32e4e9390c0a25e0af05585"
format_stamp: "Formatted at 2026-09-27 18:38:29 on dist-test-slave-9jll"
server_key: "daf22b4891bff5a777c88ff542cb8541"
server_key_iv: "881ea4f6c598647bc528405890a903fc"
server_key_version: "encryptionkey@0"
I20260927 18:38:29.513021 4881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:29.528597 4881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:29.533398 4881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:29.533723 4881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:29.553562 4881 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:46783
I20260927 18:38:29.553593 4947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:46783 every 8 connection(s)
I20260927 18:38:29.555339 4881 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:29.560010 4948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:29.561045 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 4881
I20260927 18:38:29.561309 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:29.562201 2429 external_mini_cluster.cc:1428] Setting key f0d80162bb95df8d5de2a5df68e1af6b
I20260927 18:38:29.576057 4948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585: Bootstrap starting.
Sep 27 18:38:29 dist-test-slave-9jll krb5kdc[4868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:29.580696 4948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:29.582186 4948 log.cc:824] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:29.586241 4948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585: No bootstrap required, opened a new log
I20260927 18:38:29.588824 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:29.564589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39576 (local address 127.2.95.126:46783)
0927 18:38:29.565514 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:29.565527 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:29.565578 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:38:29.567400 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:29.567431 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:29.567953 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:29.568368 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:29.570315 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.571831 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.573699 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:29.574503 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:29.574650 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:29.579859 (+ 5209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:29.579870 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:29.579889 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:29.579942 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:29.583867 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.584854 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.584863 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.584871 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.585066 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:29.585514 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:29.585518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:29.585521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:29.586228 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:29.586503 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:29.587016 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 18:38:29.587561 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":394,"threads_started":1}
I20260927 18:38:29.595373 4948 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7c197de32e4e9390c0a25e0af05585" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46783 } }
I20260927 18:38:29.595808 4948 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:29.595911 4948 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7c197de32e4e9390c0a25e0af05585, State: Initialized, Role: FOLLOWER
I20260927 18:38:29.596542 4948 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7c197de32e4e9390c0a25e0af05585" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46783 } }
I20260927 18:38:29.596761 4948 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:29.596894 4948 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:29.597049 4948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:29.599669 4948 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7c197de32e4e9390c0a25e0af05585" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46783 } }
I20260927 18:38:29.600219 4948 leader_election.cc:302] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd7c197de32e4e9390c0a25e0af05585; no voters:
I20260927 18:38:29.600860 4948 leader_election.cc:288] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:29.600993 4953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:29.601698 4953 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [term 1 LEADER]: Becoming Leader. State: Replica: cd7c197de32e4e9390c0a25e0af05585, State: Running, Role: LEADER
I20260927 18:38:29.602317 4953 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7c197de32e4e9390c0a25e0af05585" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46783 } }
I20260927 18:38:29.603080 4948 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:29.605078 4954 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd7c197de32e4e9390c0a25e0af05585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7c197de32e4e9390c0a25e0af05585" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46783 } } }
I20260927 18:38:29.605575 4954 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:29.605798 4955 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7c197de32e4e9390c0a25e0af05585. Latest consensus state: current_term: 1 leader_uuid: "cd7c197de32e4e9390c0a25e0af05585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7c197de32e4e9390c0a25e0af05585" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46783 } } }
I20260927 18:38:29.606176 4955 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:29.608117 4958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:29.613852 4958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:29.623575 4958 catalog_manager.cc:1347] Generated new cluster ID: 30d54441dffb4028b4288c25f6b967dd
I20260927 18:38:29.623710 4958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:29.638602 4958 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:29.640897 4958 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:29.660696 4958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd7c197de32e4e9390c0a25e0af05585: Generated new TSK 0
I20260927 18:38:29.662250 4958 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:38:29.723652 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:46783
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:30.113909 4974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:30.114362 4974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:30.114500 4974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:30.122654 4974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:30.122797 4974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:30.122952 4974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:38:30.134497 4974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:46783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:30.136245 4974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:30.137960 4974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:30.153214 4982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:30.153117 4979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:30.153223 4980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:30.154060 4974 server_base.cc:1034] running on GCE node
I20260927 18:38:30.154800 4974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:30.155946 4974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:30.157181 4974 hybrid_clock.cc:648] HybridClock initialized: now 1790534310157119 us; error 56 us; skew 500 ppm
I20260927 18:38:30.157550 4974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:30 dist-test-slave-9jll krb5kdc[4868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:30.161249 4974 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:38:30.162984 4974 webserver.cc:460] Webserver started at http://127.2.95.65:42623/ using document root <none> and password file <none>
I20260927 18:38:30.163869 4974 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:30.163995 4974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:30.164355 4974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:30.168140 4974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "a63e50bb80684e90978ef476c1cf2d2a"
format_stamp: "Formatted at 2026-09-27 18:38:30 on dist-test-slave-9jll"
server_key: "c1bf59ac3e7e15f6b3b21ed3222afc43"
server_key_iv: "0918fafe2a77799ef28a2bb42ef6f720"
server_key_version: "encryptionkey@0"
I20260927 18:38:30.169032 4974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "a63e50bb80684e90978ef476c1cf2d2a"
format_stamp: "Formatted at 2026-09-27 18:38:30 on dist-test-slave-9jll"
server_key: "c1bf59ac3e7e15f6b3b21ed3222afc43"
server_key_iv: "0918fafe2a77799ef28a2bb42ef6f720"
server_key_version: "encryptionkey@0"
I20260927 18:38:30.175485 4974 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260927 18:38:30.179980 4989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:30.181988 4974 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:38:30.182197 4974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "a63e50bb80684e90978ef476c1cf2d2a"
format_stamp: "Formatted at 2026-09-27 18:38:30 on dist-test-slave-9jll"
server_key: "c1bf59ac3e7e15f6b3b21ed3222afc43"
server_key_iv: "0918fafe2a77799ef28a2bb42ef6f720"
server_key_version: "encryptionkey@0"
I20260927 18:38:30.182435 4974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:30.203505 4974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:30.208834 4974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:30.209149 4974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:30.210377 4974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:30.212024 4974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:30.212110 4974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:30.212271 4974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:30.212329 4974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:30.246899 4974 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:33457
I20260927 18:38:30.246937 5101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:33457 every 8 connection(s)
I20260927 18:38:30.248425 4974 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:30.250713 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 4974
I20260927 18:38:30.250957 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:30.251845 2429 external_mini_cluster.cc:1428] Setting key eb95738614543fdc999834f90800d669
Sep 27 18:38:30 dist-test-slave-9jll krb5kdc[4868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
I20260927 18:38:30.274511 5104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:30.251731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:58489 (local address 127.2.95.126:46783)
0927 18:38:30.252248 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:30.252253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:30.253628 (+ 1375us) server_negotiation.cc:407] Connection header received
0927 18:38:30.256355 (+ 2727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:30.256360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:30.256438 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:30.256536 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:30.258906 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.259758 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.261959 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.262588 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.262750 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:30.266953 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:30.266960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:30.266965 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:30.267009 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:30.270720 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:30.271639 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:30.271645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:30.271649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:30.271822 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:30.272264 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:30.272269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:30.272272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:30.272748 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:30.272911 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:30.273856 (+ 945us) server_negotiation.cc:299] Negotiation successful
0927 18:38:30.274123 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":260,"threads_started":1}
I20260927 18:38:30.275579 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:30.252505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46783 (local address 127.2.95.65:58489)
0927 18:38:30.253418 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:30.253485 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:30.255895 (+ 2410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:30.256878 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:30.256892 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:30.257748 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:30.258699 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:30.258728 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.259918 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.259923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:30.261777 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:30.261790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.262842 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.262847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:30.263083 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:30.263895 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:30.263926 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:30.266566 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:30.270916 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:30.270926 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:30.270946 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:30.271482 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:30.271929 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:30.271933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:30.271935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:30.272142 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:30.272906 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:30.272914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:30.273364 (+ 450us) client_negotiation.cc:769] Sending connection context
0927 18:38:30.273656 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 18:38:30.274231 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":348,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
I20260927 18:38:30.277782 5102 heartbeater.cc:347] Connected to a master server at 127.2.95.126:46783
I20260927 18:38:30.278143 5102 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:30.279055 5102 heartbeater.cc:511] Master 127.2.95.126:46783 requested a full tablet report, sending...
I20260927 18:38:30.281957 4913 ts_manager.cc:194] Registered new tserver with Master: a63e50bb80684e90978ef476c1cf2d2a (127.2.95.65:33457)
I20260927 18:38:30.283716 4913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:58489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:38:30.305094 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:46783
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:30.683476 5108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:30.683828 5108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:30.683933 5108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:30.692046 5108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:30.692200 5108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:30.692327 5108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:38:30.703380 5108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:46783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:30.705139 5108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:30.706810 5108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:30.721697 5113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:30.721726 5116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:30.721639 5114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:30.722664 5108 server_base.cc:1034] running on GCE node
I20260927 18:38:30.723482 5108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:30.724890 5108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:30.726195 5108 hybrid_clock.cc:648] HybridClock initialized: now 1790534310726117 us; error 64 us; skew 500 ppm
I20260927 18:38:30.726593 5108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:30 dist-test-slave-9jll krb5kdc[4868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:30.730309 5108 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:38:30.732177 5108 webserver.cc:460] Webserver started at http://127.2.95.66:37171/ using document root <none> and password file <none>
I20260927 18:38:30.733039 5108 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:30.733157 5108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:30.733507 5108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:30.736999 5108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "2a325b07890e42fc991d99c098a33fa2"
format_stamp: "Formatted at 2026-09-27 18:38:30 on dist-test-slave-9jll"
server_key: "d55907fc0e75811234d2fdb834fd4a6f"
server_key_iv: "1a0376b5278b00cf00cab60e5bcd34a3"
server_key_version: "encryptionkey@0"
I20260927 18:38:30.737920 5108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "2a325b07890e42fc991d99c098a33fa2"
format_stamp: "Formatted at 2026-09-27 18:38:30 on dist-test-slave-9jll"
server_key: "d55907fc0e75811234d2fdb834fd4a6f"
server_key_iv: "1a0376b5278b00cf00cab60e5bcd34a3"
server_key_version: "encryptionkey@0"
I20260927 18:38:30.743988 5108 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:38:30.748149 5123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:30.749907 5108 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:38:30.750089 5108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "2a325b07890e42fc991d99c098a33fa2"
format_stamp: "Formatted at 2026-09-27 18:38:30 on dist-test-slave-9jll"
server_key: "d55907fc0e75811234d2fdb834fd4a6f"
server_key_iv: "1a0376b5278b00cf00cab60e5bcd34a3"
server_key_version: "encryptionkey@0"
I20260927 18:38:30.750334 5108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:30.767586 5108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:30.772742 5108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:30.773070 5108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:30.774258 5108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:30.775811 5108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:30.775898 5108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:30.776043 5108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:30.776113 5108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:30.812669 5108 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:35793
I20260927 18:38:30.812709 5235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:35793 every 8 connection(s)
I20260927 18:38:30.814244 5108 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:30.818550 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 5108
I20260927 18:38:30.818768 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:30.819404 2429 external_mini_cluster.cc:1428] Setting key ff732dd6245fab381ef8d7921ed76045
Sep 27 18:38:30 dist-test-slave-9jll krb5kdc[4868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:30.841472 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:30.817426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:37497 (local address 127.2.95.126:46783)
0927 18:38:30.818047 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:30.818052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:30.819512 (+ 1460us) server_negotiation.cc:407] Connection header received
0927 18:38:30.821999 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:30.822006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:30.822109 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:30.822252 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:30.824982 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.825926 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.828423 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.828992 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.829171 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:30.834218 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:30.834226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:30.834230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:30.834279 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:30.837563 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:30.838614 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:30.838620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:30.838624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:30.838806 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:30.839283 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:30.839288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:30.839292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:30.839698 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:30.839870 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:30.840741 (+ 871us) server_negotiation.cc:299] Negotiation successful
0927 18:38:30.841036 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":296,"threads_started":1}
I20260927 18:38:30.842657 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:30.818223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46783 (local address 127.2.95.66:37497)
0927 18:38:30.819290 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:30.819347 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:30.821631 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:30.822626 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:30.822641 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:30.823607 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:30.824715 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:30.824747 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.826129 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.826135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:30.828217 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:30.828235 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:30.829317 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:30.829324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:30.829711 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:30.830654 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:30.830686 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:30.833873 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:30.837795 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:30.837805 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:30.837826 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:30.838448 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:30.838922 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:30.838926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:30.838928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:30.839163 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:30.839841 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:30.839849 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:30.840330 (+ 481us) client_negotiation.cc:769] Sending connection context
0927 18:38:30.840614 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 18:38:30.841155 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":448,"threads_started":1}
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
I20260927 18:38:30.844439 5236 heartbeater.cc:347] Connected to a master server at 127.2.95.126:46783
I20260927 18:38:30.844825 5236 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
I20260927 18:38:30.845887 5236 heartbeater.cc:511] Master 127.2.95.126:46783 requested a full tablet report, sending...
I20260927 18:38:30.847764 4913 ts_manager.cc:194] Registered new tserver with Master: 2a325b07890e42fc991d99c098a33fa2 (127.2.95.66:35793)
I20260927 18:38:30.848701 4913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:37497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:38:30.873755 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:46783
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:31.253859 5242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:31.254263 5242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:31.254412 5242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:31.262521 5242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:31.262655 5242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:31.262813 5242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:38:31.273770 5242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:46783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:31.275530 5242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:31.277164 5242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:38:31.288048 5102 heartbeater.cc:503] Master 127.2.95.126:46783 was elected leader, sending a full tablet report...
W20260927 18:38:31.292934 5250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:31.292958 5248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:31.293592 5247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:31.293915 5242 server_base.cc:1034] running on GCE node
I20260927 18:38:31.294687 5242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:31.295862 5242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:31.297178 5242 hybrid_clock.cc:648] HybridClock initialized: now 1790534311297135 us; error 39 us; skew 500 ppm
I20260927 18:38:31.297559 5242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:31 dist-test-slave-9jll krb5kdc[4868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:31.301009 5242 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:38:31.302822 5242 webserver.cc:460] Webserver started at http://127.2.95.67:34715/ using document root <none> and password file <none>
I20260927 18:38:31.303715 5242 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:31.303838 5242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:31.304191 5242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:31.307734 5242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "a03c8a53202549a58ca06989ec5b2547"
format_stamp: "Formatted at 2026-09-27 18:38:31 on dist-test-slave-9jll"
server_key: "6fd97adb9817f2a5960b401de902c09c"
server_key_iv: "478c23ce8c4b1b7f472e74cb4db6338a"
server_key_version: "encryptionkey@0"
I20260927 18:38:31.308653 5242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "a03c8a53202549a58ca06989ec5b2547"
format_stamp: "Formatted at 2026-09-27 18:38:31 on dist-test-slave-9jll"
server_key: "6fd97adb9817f2a5960b401de902c09c"
server_key_iv: "478c23ce8c4b1b7f472e74cb4db6338a"
server_key_version: "encryptionkey@0"
I20260927 18:38:31.315014 5242 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 18:38:31.319248 5257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:31.321069 5242 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:38:31.321250 5242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "a03c8a53202549a58ca06989ec5b2547"
format_stamp: "Formatted at 2026-09-27 18:38:31 on dist-test-slave-9jll"
server_key: "6fd97adb9817f2a5960b401de902c09c"
server_key_iv: "478c23ce8c4b1b7f472e74cb4db6338a"
server_key_version: "encryptionkey@0"
I20260927 18:38:31.321471 5242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:31.338351 5242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:31.343662 5242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:31.343987 5242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:31.345122 5242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:31.346807 5242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:31.346892 5242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:31.347026 5242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:31.347085 5242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:31.383384 5242 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:36095
I20260927 18:38:31.383473 5369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:36095 every 8 connection(s)
I20260927 18:38:31.384963 5242 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:38:31.386960 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 5242
I20260927 18:38:31.387189 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:31.387853 2429 external_mini_cluster.cc:1428] Setting key 45f350f1b23dd88fbc216a37c328eab6
Sep 27 18:38:31 dist-test-slave-9jll krb5kdc[4868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:31.409200 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.388460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:38583 (local address 127.2.95.126:46783)
0927 18:38:31.389533 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.389540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.390510 (+ 970us) server_negotiation.cc:407] Connection header received
0927 18:38:31.393361 (+ 2851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.393366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.393446 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.393562 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.395430 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.396291 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.398117 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.398572 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.398684 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.402972 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:31.402978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:31.402981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:31.403021 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:31.405978 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.406732 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.406737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.406739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.406871 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:31.407261 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:31.407265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:31.407267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:31.407550 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:31.407685 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:31.408648 (+ 963us) server_negotiation.cc:299] Negotiation successful
0927 18:38:31.408837 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":793,"threads_started":1}
I20260927 18:38:31.410530 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:31.389325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46783 (local address 127.2.95.67:38583)
0927 18:38:31.390333 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:31.390395 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:31.393016 (+ 2621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:31.393885 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:31.393896 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:31.394554 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:31.395258 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.395276 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.396427 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.396432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.397942 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:31.397955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:31.398815 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:31.398821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:31.399050 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:31.399864 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:31.399890 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:31.402728 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:31.406113 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.406121 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.406139 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.406616 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.406949 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:31.406952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:31.406955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:31.407164 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:31.407704 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:31.407712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:31.408161 (+ 449us) client_negotiation.cc:769] Sending connection context
0927 18:38:31.408456 (+ 295us) client_negotiation.cc:240] Negotiation successful
0927 18:38:31.408992 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":422,"threads_started":1}
I20260927 18:38:31.412231 5370 heartbeater.cc:347] Connected to a master server at 127.2.95.126:46783
I20260927 18:38:31.412621 5370 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:31.413621 5370 heartbeater.cc:511] Master 127.2.95.126:46783 requested a full tablet report, sending...
I20260927 18:38:31.415630 4913 ts_manager.cc:194] Registered new tserver with Master: a03c8a53202549a58ca06989ec5b2547 (127.2.95.67:36095)
I20260927 18:38:31.416839 4913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:38583
I20260927 18:38:31.417448 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:38:31.452090 5372 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:31.449365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39586 (local address 127.2.95.126:46783)
0927 18:38:31.449583 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:31.449588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:31.450014 (+ 426us) server_negotiation.cc:407] Connection header received
0927 18:38:31.450188 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:31.450192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:31.450272 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:31.450374 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:31.451791 (+ 1417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39586: BlockingRecv error: recv got EOF from 127.0.0.1:39586 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:38:31.455408 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 4974
W20260927 18:38:31.458590 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 4974: Not authorized: Client connection negotiation failed: client connection to 127.2.95.65:33457: server requires authentication, but client does not have Kerberos credentials available
I20260927 18:38:31.458775 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 4974
I20260927 18:38:31.473191 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 5108
W20260927 18:38:31.476593 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 5108: Not authorized: Client connection negotiation failed: client connection to 127.2.95.66:35793: server requires authentication, but client does not have Kerberos credentials available
I20260927 18:38:31.476801 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 5108
I20260927 18:38:31.491499 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 5242
W20260927 18:38:31.495299 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 5242: Not authorized: Client connection negotiation failed: client connection to 127.2.95.67:36095: server requires authentication, but client does not have Kerberos credentials available
I20260927 18:38:31.495476 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 5242
I20260927 18:38:31.510254 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 4881
I20260927 18:38:31.511859 4943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:31.625494 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 4881
2026-09-27T18:38:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3778 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:31 dist-test-slave-9jll krb5kdc[5388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:31 dist-test-slave-9jll krb5kdc[5388](info): set up 2 sockets
Sep 27 18:38:31 dist-test-slave-9jll krb5kdc[5388](info): commencing operation
krb5kdc: starting...
W20260927 18:38:32.655855 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:38:32 dist-test-slave-9jll krb5kdc[5388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:38:32.879738 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33471
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:34897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:33471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:33.263051 5401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:33.263438 5401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:33.263562 5401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:33.271647 5401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:33.271793 5401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:33.271852 5401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:33.271939 5401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:33.272006 5401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:33.283416 5401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:34897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:33471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:33.285202 5401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:33.286907 5401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:33.300168 5406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:33.301050 5409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:33.301071 5407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:33.301106 5401 server_base.cc:1034] running on GCE node
I20260927 18:38:33.302280 5401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:33.303997 5401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:33.305246 5401 hybrid_clock.cc:648] HybridClock initialized: now 1790534313305184 us; error 51 us; skew 500 ppm
I20260927 18:38:33.305696 5401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:33 dist-test-slave-9jll krb5kdc[5388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534313, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:33.316519 5401 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:38:33.318490 5401 webserver.cc:460] Webserver started at http://127.2.95.126:44499/ using document root <none> and password file <none>
I20260927 18:38:33.319355 5401 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:33.319509 5401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:33.319898 5401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:33.323575 5401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "c5923bcfdd4843869df2b49c9c005078"
format_stamp: "Formatted at 2026-09-27 18:38:33 on dist-test-slave-9jll"
server_key: "32929a38df80d40b50b06966f1f880a5"
server_key_iv: "4561085fd1e8b49856b4fa330cd99bb1"
server_key_version: "encryptionkey@0"
I20260927 18:38:33.324429 5401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "c5923bcfdd4843869df2b49c9c005078"
format_stamp: "Formatted at 2026-09-27 18:38:33 on dist-test-slave-9jll"
server_key: "32929a38df80d40b50b06966f1f880a5"
server_key_iv: "4561085fd1e8b49856b4fa330cd99bb1"
server_key_version: "encryptionkey@0"
I20260927 18:38:33.330588 5401 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 18:38:33.334818 5416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:33.336544 5401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260927 18:38:33.336722 5401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "c5923bcfdd4843869df2b49c9c005078"
format_stamp: "Formatted at 2026-09-27 18:38:33 on dist-test-slave-9jll"
server_key: "32929a38df80d40b50b06966f1f880a5"
server_key_iv: "4561085fd1e8b49856b4fa330cd99bb1"
server_key_version: "encryptionkey@0"
I20260927 18:38:33.336941 5401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:33.351663 5401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:33.357360 5401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:33.357753 5401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:33.377172 5401 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:33471
I20260927 18:38:33.377204 5467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:33471 every 8 connection(s)
I20260927 18:38:33.378780 5401 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:33.383322 5468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:33.384298 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 5401
I20260927 18:38:33.384514 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:33.385357 2429 external_mini_cluster.cc:1428] Setting key 18b8b012f5aafe217a9a434cdbd2aa8f
I20260927 18:38:33.399531 5468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078: Bootstrap starting.
Sep 27 18:38:33 dist-test-slave-9jll krb5kdc[5388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:33.403362 5468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:33.404927 5468 log.cc:824] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:33.408874 5468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078: No bootstrap required, opened a new log
I20260927 18:38:33.412660 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:33.387697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.2.95.126:33471)
0927 18:38:33.388713 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:33.388731 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:33.388807 (+ 76us) server_negotiation.cc:407] Connection header received
0927 18:38:33.391010 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:33.391051 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:33.391748 (+ 697us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:33.392325 (+ 577us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:33.393971 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.395986 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.397704 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.398813 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.399020 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:33.403480 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:33.403493 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:33.403516 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:33.403571 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:33.407290 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:33.408238 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:33.408247 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:33.408256 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:33.408480 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:33.408988 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:33.408993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:33.408996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:33.409848 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:33.410156 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:33.410608 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 18:38:33.411113 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":358,"threads_started":1}
I20260927 18:38:33.417997 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:33.419209 5468 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5923bcfdd4843869df2b49c9c005078" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33471 } }
I20260927 18:38:33.419716 5468 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:33.419814 5468 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5923bcfdd4843869df2b49c9c005078, State: Initialized, Role: FOLLOWER
I20260927 18:38:33.420555 5468 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5923bcfdd4843869df2b49c9c005078" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33471 } }
I20260927 18:38:33.420831 5468 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:33.420960 5468 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:33.421128 5468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:33.424170 5468 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5923bcfdd4843869df2b49c9c005078" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33471 } }
I20260927 18:38:33.424870 5468 leader_election.cc:302] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5923bcfdd4843869df2b49c9c005078; no voters:
I20260927 18:38:33.425570 5468 leader_election.cc:288] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:33.425709 5473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:33.426934 5473 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [term 1 LEADER]: Becoming Leader. State: Replica: c5923bcfdd4843869df2b49c9c005078, State: Running, Role: LEADER
I20260927 18:38:33.427589 5473 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5923bcfdd4843869df2b49c9c005078" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33471 } }
I20260927 18:38:33.428117 5468 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:33.431623 5474 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5923bcfdd4843869df2b49c9c005078" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5923bcfdd4843869df2b49c9c005078" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33471 } } }
I20260927 18:38:33.431604 5475 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5923bcfdd4843869df2b49c9c005078. Latest consensus state: current_term: 1 leader_uuid: "c5923bcfdd4843869df2b49c9c005078" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5923bcfdd4843869df2b49c9c005078" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33471 } } }
I20260927 18:38:33.432134 5474 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:33.432145 5475 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:33.443987 5481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:33.451529 5481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:33.475696 5481 catalog_manager.cc:1347] Generated new cluster ID: 31c1207333454d76aa739ab55e10ff6c
I20260927 18:38:33.475842 5481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:33.494174 5481 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:33.496120 5481 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:33.511379 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:33.492020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.2.95.126:33471)
0927 18:38:33.492189 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:33.492195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:33.502378 (+ 10183us) server_negotiation.cc:407] Connection header received
0927 18:38:33.502673 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:33.502680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:33.502768 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:33.502837 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:33.504348 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.505506 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.507458 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:33.508153 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:33.508321 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:33.509742 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:33.509750 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:33.509755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:33.510507 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:33.510591 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:33.510627 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:33.510741 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:33.510914 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 18:38:33.511016 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:38:33.512176 5481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5923bcfdd4843869df2b49c9c005078: Generated new TSK 0
I20260927 18:38:33.513278 5481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:33.522894 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 5401
I20260927 18:38:33.524175 5463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:33.615449 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 5401
2026-09-27T18:38:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1982 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:33 dist-test-slave-9jll krb5kdc[5502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:33 dist-test-slave-9jll krb5kdc[5502](info): set up 2 sockets
Sep 27 18:38:33 dist-test-slave-9jll krb5kdc[5502](info): commencing operation
krb5kdc: starting...
W20260927 18:38:34.623035 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:38:34 dist-test-slave-9jll krb5kdc[5502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:38:34.818337 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:37563
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:45715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:37563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:35.167953 5515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:35.168268 5515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:35.168358 5515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:35.175845 5515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:35.175976 5515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:35.176020 5515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:35.176092 5515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:35.176151 5515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:35.186112 5515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:45715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:37563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:37563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:35.187500 5515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:35.188992 5515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:35.201110 5523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:35.201076 5520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:35.201149 5521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:35.202443 5515 server_base.cc:1034] running on GCE node
I20260927 18:38:35.203321 5515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:35.204952 5515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:35.206207 5515 hybrid_clock.cc:648] HybridClock initialized: now 1790534315206134 us; error 60 us; skew 500 ppm
I20260927 18:38:35.206621 5515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:35 dist-test-slave-9jll krb5kdc[5502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:35.209864 5515 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:38:35.211452 5515 webserver.cc:460] Webserver started at http://127.2.95.126:45787/ using document root <none> and password file <none>
I20260927 18:38:35.212153 5515 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:35.212263 5515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:35.212544 5515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:35.215742 5515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "b5f3268be5c24915be6f11e0d0d7912c"
format_stamp: "Formatted at 2026-09-27 18:38:35 on dist-test-slave-9jll"
server_key: "038b71a6f0d6cb2490144fc9921cb590"
server_key_iv: "b428e1af8b7b1bb2e3cd022268ca84ad"
server_key_version: "encryptionkey@0"
I20260927 18:38:35.216540 5515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "b5f3268be5c24915be6f11e0d0d7912c"
format_stamp: "Formatted at 2026-09-27 18:38:35 on dist-test-slave-9jll"
server_key: "038b71a6f0d6cb2490144fc9921cb590"
server_key_iv: "b428e1af8b7b1bb2e3cd022268ca84ad"
server_key_version: "encryptionkey@0"
I20260927 18:38:35.222078 5515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 18:38:35.225857 5530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:35.227355 5515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:38:35.227512 5515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "b5f3268be5c24915be6f11e0d0d7912c"
format_stamp: "Formatted at 2026-09-27 18:38:35 on dist-test-slave-9jll"
server_key: "038b71a6f0d6cb2490144fc9921cb590"
server_key_iv: "b428e1af8b7b1bb2e3cd022268ca84ad"
server_key_version: "encryptionkey@0"
I20260927 18:38:35.227711 5515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:35.248589 5515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:35.253374 5515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:35.253661 5515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:35.270700 5515 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:37563
I20260927 18:38:35.270723 5581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:37563 every 8 connection(s)
I20260927 18:38:35.272078 5515 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:35.276108 5582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:35.280082 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 5515
I20260927 18:38:35.280277 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:35.280922 2429 external_mini_cluster.cc:1428] Setting key 29a15b8cdafce10eba3e65e3b8369fba
I20260927 18:38:35.287431 5582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c: Bootstrap starting.
I20260927 18:38:35.291651 5582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:35.292843 5582 log.cc:824] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:38:35 dist-test-slave-9jll krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:35.296392 5582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c: No bootstrap required, opened a new log
I20260927 18:38:35.304352 5582 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f3268be5c24915be6f11e0d0d7912c" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37563 } }
I20260927 18:38:35.304764 5582 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:35.304857 5582 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f3268be5c24915be6f11e0d0d7912c, State: Initialized, Role: FOLLOWER
I20260927 18:38:35.305145 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:35.282939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.2.95.126:37563)
0927 18:38:35.283707 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:35.283719 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:35.283760 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:38:35.285908 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:35.285936 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:35.286366 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:35.286709 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:35.288463 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:35.289930 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:35.291914 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:35.292656 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:35.292793 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:35.297350 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:35.297359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:35.297376 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:35.297419 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:35.300782 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:35.301576 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:35.301583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:35.301593 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:35.301812 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:35.302197 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:35.302202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:35.302205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:35.302818 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:35.303128 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:35.303562 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 18:38:35.304001 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":348,"threads_started":1}
I20260927 18:38:35.305495 5582 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f3268be5c24915be6f11e0d0d7912c" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37563 } }
I20260927 18:38:35.305809 5582 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:35.305943 5582 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:35.306116 5582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:35.308640 5582 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f3268be5c24915be6f11e0d0d7912c" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37563 } }
I20260927 18:38:35.309083 5582 leader_election.cc:302] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5f3268be5c24915be6f11e0d0d7912c; no voters:
I20260927 18:38:35.309657 5582 leader_election.cc:288] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:35.309825 5587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:35.311244 5587 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [term 1 LEADER]: Becoming Leader. State: Replica: b5f3268be5c24915be6f11e0d0d7912c, State: Running, Role: LEADER
I20260927 18:38:35.311815 5587 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f3268be5c24915be6f11e0d0d7912c" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37563 } }
I20260927 18:38:35.312391 5582 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:35.314924 5589 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5f3268be5c24915be6f11e0d0d7912c. Latest consensus state: current_term: 1 leader_uuid: "b5f3268be5c24915be6f11e0d0d7912c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f3268be5c24915be6f11e0d0d7912c" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37563 } } }
I20260927 18:38:35.315317 5589 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:35.315768 5588 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5f3268be5c24915be6f11e0d0d7912c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f3268be5c24915be6f11e0d0d7912c" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37563 } } }
I20260927 18:38:35.316071 5588 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:35.327039 5594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:35.331694 5594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:35.340781 5594 catalog_manager.cc:1347] Generated new cluster ID: 5505b0232ef04a70957395c8d004d902
I20260927 18:38:35.340926 5594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:35.357568 5594 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:35.359153 5594 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:35.369491 5594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5f3268be5c24915be6f11e0d0d7912c: Generated new TSK 0
I20260927 18:38:35.370573 5594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:38:35.425416 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37563
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:45715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:35.754922 5608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:35.755218 5608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:35.755312 5608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:35.762874 5608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:35.762992 5608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:35.763106 5608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:38:35.773183 5608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:45715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:35.774454 5608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:35.775691 5608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:35.788997 5613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:35.789032 5616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:35.788996 5614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:35.789908 5608 server_base.cc:1034] running on GCE node
I20260927 18:38:35.790525 5608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:35.791580 5608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:35.792773 5608 hybrid_clock.cc:648] HybridClock initialized: now 1790534315792734 us; error 34 us; skew 500 ppm
I20260927 18:38:35.793078 5608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:35 dist-test-slave-9jll krb5kdc[5502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:35.796355 5608 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:38:35.797924 5608 webserver.cc:460] Webserver started at http://127.2.95.65:42143/ using document root <none> and password file <none>
I20260927 18:38:35.798614 5608 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:35.798712 5608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:35.798985 5608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:35.802052 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "dcc2855441a04d24b13ac71c25e7ce69"
format_stamp: "Formatted at 2026-09-27 18:38:35 on dist-test-slave-9jll"
server_key: "72fec4b10daea3d163ab64f0cd96199d"
server_key_iv: "362926e3422ead699668982ba8819747"
server_key_version: "encryptionkey@0"
I20260927 18:38:35.802799 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "dcc2855441a04d24b13ac71c25e7ce69"
format_stamp: "Formatted at 2026-09-27 18:38:35 on dist-test-slave-9jll"
server_key: "72fec4b10daea3d163ab64f0cd96199d"
server_key_iv: "362926e3422ead699668982ba8819747"
server_key_version: "encryptionkey@0"
I20260927 18:38:35.808611 5608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:38:35.812885 5623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:35.814566 5608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:38:35.814713 5608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "dcc2855441a04d24b13ac71c25e7ce69"
format_stamp: "Formatted at 2026-09-27 18:38:35 on dist-test-slave-9jll"
server_key: "72fec4b10daea3d163ab64f0cd96199d"
server_key_iv: "362926e3422ead699668982ba8819747"
server_key_version: "encryptionkey@0"
I20260927 18:38:35.814901 5608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:35.835035 5608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:35.839674 5608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:35.839948 5608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:35.841054 5608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:35.842645 5608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:35.842731 5608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:35.842826 5608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:35.842870 5608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:35.873987 5608 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:45885
I20260927 18:38:35.874044 5735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:45885 every 8 connection(s)
I20260927 18:38:35.875339 5608 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:35.885442 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 5608
I20260927 18:38:35.885638 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:35.886300 2429 external_mini_cluster.cc:1428] Setting key 58d4ee9b278489fb49814edae7bc33b7
Sep 27 18:38:35 dist-test-slave-9jll krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:35.896651 5738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:35.877955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:39113 (local address 127.2.95.126:37563)
0927 18:38:35.878372 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:35.878376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:35.879420 (+ 1044us) server_negotiation.cc:407] Connection header received
0927 18:38:35.881262 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:35.881269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:35.881352 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:35.881445 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:35.883107 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:35.883830 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:35.885425 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:35.885920 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:35.886034 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:35.890643 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:35.890649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:35.890652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:35.890689 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:35.893309 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:35.894234 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:35.894239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:35.894241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:35.894374 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:35.894830 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:35.894834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:35.894838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:35.895140 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:35.895275 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:35.896057 (+ 782us) server_negotiation.cc:299] Negotiation successful
0927 18:38:35.896290 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":207,"threads_started":1}
I20260927 18:38:35.897745 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:35.878463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37563 (local address 127.2.95.65:39113)
0927 18:38:35.879195 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:35.879240 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:35.880961 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:35.881748 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:35.881759 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:35.882305 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:35.882908 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:35.882927 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:35.883991 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:35.883995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:35.885271 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:35.885284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:35.886151 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:35.886158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:35.886394 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:35.887217 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:35.887245 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:35.890347 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:35.893520 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:35.893530 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:35.893546 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:35.894088 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:35.894493 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:35.894497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:35.894499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:35.894704 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:35.895274 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:35.895283 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:35.895665 (+ 382us) client_negotiation.cc:769] Sending connection context
0927 18:38:35.895926 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 18:38:35.896507 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":288,"threads_started":1}
I20260927 18:38:35.899389 5736 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37563
I20260927 18:38:35.899678 5736 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:35.900439 5736 heartbeater.cc:511] Master 127.2.95.126:37563 requested a full tablet report, sending...
I20260927 18:38:35.902786 5547 ts_manager.cc:194] Registered new tserver with Master: dcc2855441a04d24b13ac71c25e7ce69 (127.2.95.65:45885)
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
I20260927 18:38:35.904312 5547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:39113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:38:35.930197 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37563
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:45715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:36.256332 5742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:36.256695 5742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:36.256798 5742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:36.264307 5742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:36.264441 5742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:36.264557 5742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:38:36.277601 5742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:45715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:36.279202 5742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:36.280906 5742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:36.295540 5747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:36.295578 5750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:36.295605 5748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:36.296700 5742 server_base.cc:1034] running on GCE node
I20260927 18:38:36.297346 5742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:36.298424 5742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:36.299628 5742 hybrid_clock.cc:648] HybridClock initialized: now 1790534316299569 us; error 52 us; skew 500 ppm
I20260927 18:38:36.300031 5742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:36 dist-test-slave-9jll krb5kdc[5502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534316, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:36.303504 5742 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:38:36.305135 5742 webserver.cc:460] Webserver started at http://127.2.95.66:40621/ using document root <none> and password file <none>
I20260927 18:38:36.305909 5742 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:36.306028 5742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:36.306313 5742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:36.309274 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "6af254e0074f4e90987de63056827dd9"
format_stamp: "Formatted at 2026-09-27 18:38:36 on dist-test-slave-9jll"
server_key: "5c68676509b9fa4d1a1e5c1ec3d3afb2"
server_key_iv: "966b5d52c80d92722822ab87c896951e"
server_key_version: "encryptionkey@0"
I20260927 18:38:36.310101 5742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "6af254e0074f4e90987de63056827dd9"
format_stamp: "Formatted at 2026-09-27 18:38:36 on dist-test-slave-9jll"
server_key: "5c68676509b9fa4d1a1e5c1ec3d3afb2"
server_key_iv: "966b5d52c80d92722822ab87c896951e"
server_key_version: "encryptionkey@0"
I20260927 18:38:36.315567 5742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:38:36.319399 5757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:36.321197 5742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 18:38:36.321436 5742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "6af254e0074f4e90987de63056827dd9"
format_stamp: "Formatted at 2026-09-27 18:38:36 on dist-test-slave-9jll"
server_key: "5c68676509b9fa4d1a1e5c1ec3d3afb2"
server_key_iv: "966b5d52c80d92722822ab87c896951e"
server_key_version: "encryptionkey@0"
I20260927 18:38:36.321708 5742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:36.347234 5742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:36.351883 5742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:36.352180 5742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:36.353216 5742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:36.354717 5742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:36.354795 5742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:36.354907 5742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:36.354956 5742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:36.386626 5742 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:41929
I20260927 18:38:36.386667 5869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:41929 every 8 connection(s)
I20260927 18:38:36.387907 5742 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:36.389473 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 5742
I20260927 18:38:36.389621 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:36.390201 2429 external_mini_cluster.cc:1428] Setting key 76424d4f2393d06730347634e9f98598
Sep 27 18:38:36 dist-test-slave-9jll krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534316, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
I20260927 18:38:36.409927 5738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:36.390668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:35247 (local address 127.2.95.126:37563)
0927 18:38:36.391047 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:36.391052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:36.392205 (+ 1153us) server_negotiation.cc:407] Connection header received
0927 18:38:36.394742 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:36.394747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:36.394900 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:36.394990 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:36.397026 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:36.397892 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:36.399565 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:36.400007 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:36.400123 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:36.404026 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:36.404032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:36.404035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:36.404072 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:36.406825 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:36.407598 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:36.407602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:36.407605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:36.407732 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:36.408191 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:36.408195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:36.408197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:36.408465 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:36.408593 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:36.409294 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 18:38:36.409522 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
I20260927 18:38:36.410941 5872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:36.391205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37563 (local address 127.2.95.66:35247)
0927 18:38:36.392030 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:36.392089 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:36.394405 (+ 2316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:36.395277 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:36.395290 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:36.396001 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:36.396798 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:36.396823 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:36.398072 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:36.398077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:36.399405 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:36.399420 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:36.400255 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:36.400263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:36.400491 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:36.401220 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:36.401244 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:36.403705 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:36.406996 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:36.407007 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:36.407023 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:36.407472 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:36.407851 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:36.407856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:36.407859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:36.408082 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:36.408589 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:36.408598 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:36.408949 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 18:38:36.409197 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 18:38:36.409823 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":295,"threads_started":1}
I20260927 18:38:36.412462 5870 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37563
I20260927 18:38:36.412784 5870 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:36.413614 5870 heartbeater.cc:511] Master 127.2.95.126:37563 requested a full tablet report, sending...
I20260927 18:38:36.415148 5547 ts_manager.cc:194] Registered new tserver with Master: 6af254e0074f4e90987de63056827dd9 (127.2.95.66:41929)
I20260927 18:38:36.416013 5547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:35247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:38:36.436846 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37563
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:45715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:36.763906 5875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:36.764212 5875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:36.764349 5875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:36.772060 5875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:36.772212 5875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:36.772370 5875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:38:36.783052 5875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:45715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:36.784423 5875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:36.785759 5875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:36.799242 5881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:36.799130 5883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:36.799145 5880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:36.800318 5875 server_base.cc:1034] running on GCE node
I20260927 18:38:36.800961 5875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:36.802014 5875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:36.803246 5875 hybrid_clock.cc:648] HybridClock initialized: now 1790534316803159 us; error 75 us; skew 500 ppm
I20260927 18:38:36.803591 5875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:36 dist-test-slave-9jll krb5kdc[5502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534316, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:36.807184 5875 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:38:36.808828 5875 webserver.cc:460] Webserver started at http://127.2.95.67:39799/ using document root <none> and password file <none>
I20260927 18:38:36.809585 5875 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:36.809748 5875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:36.810062 5875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:36.813313 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "ace6fe91e1004255b553d36df9dd297e"
format_stamp: "Formatted at 2026-09-27 18:38:36 on dist-test-slave-9jll"
server_key: "0cdbec544c3c39b19e70d389c29665fc"
server_key_iv: "ea43fa5cad7bd98e8c2f8e47bc75461e"
server_key_version: "encryptionkey@0"
I20260927 18:38:36.814221 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "ace6fe91e1004255b553d36df9dd297e"
format_stamp: "Formatted at 2026-09-27 18:38:36 on dist-test-slave-9jll"
server_key: "0cdbec544c3c39b19e70d389c29665fc"
server_key_iv: "ea43fa5cad7bd98e8c2f8e47bc75461e"
server_key_version: "encryptionkey@0"
I20260927 18:38:36.820241 5875 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 18:38:36.824411 5890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:36.826265 5875 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:38:36.826471 5875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "ace6fe91e1004255b553d36df9dd297e"
format_stamp: "Formatted at 2026-09-27 18:38:36 on dist-test-slave-9jll"
server_key: "0cdbec544c3c39b19e70d389c29665fc"
server_key_iv: "ea43fa5cad7bd98e8c2f8e47bc75461e"
server_key_version: "encryptionkey@0"
I20260927 18:38:36.826684 5875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:36.844378 5875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:36.848759 5875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:36.849047 5875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:36.850162 5875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:36.851792 5875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:36.851872 5875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:36.851995 5875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:36.852047 5875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:36.883585 5875 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:45073
I20260927 18:38:36.883628 6002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:45073 every 8 connection(s)
I20260927 18:38:36.884827 5875 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:38:36.885824 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 5875
I20260927 18:38:36.886000 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:36.886512 2429 external_mini_cluster.cc:1428] Setting key 26f1c67e6616139bb45af9a3e8bc4fd6
Sep 27 18:38:36 dist-test-slave-9jll krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534316, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:36.905516 5738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:36.887614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:60951 (local address 127.2.95.126:37563)
0927 18:38:36.887794 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:36.887800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:36.889225 (+ 1425us) server_negotiation.cc:407] Connection header received
0927 18:38:36.891189 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:36.891195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:36.891286 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:36.891366 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:36.892958 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:36.893775 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:36.895348 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:36.895754 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:36.895867 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:36.899621 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:36.899626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:36.899630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:36.899667 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:36.902405 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:36.903165 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:36.903170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:36.903172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:36.903396 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:36.903850 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:36.903854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:36.903856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:36.904138 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:36.904258 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:36.905069 (+ 811us) server_negotiation.cc:299] Negotiation successful
0927 18:38:36.905251 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:38:36.906476 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:36.888161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37563 (local address 127.2.95.67:60951)
0927 18:38:36.889075 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:36.889113 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:36.890925 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:36.891630 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:36.891640 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:36.892167 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:36.892778 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:36.892796 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:36.893931 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:36.893935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:36.895197 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:36.895210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:36.895981 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:36.895988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:36.896235 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:36.896891 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:36.896916 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:36.899358 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:36.902553 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:36.902562 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:36.902578 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:36.903038 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:36.903517 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:36.903521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:36.903523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:36.903724 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:36.904263 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:36.904270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:36.904623 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 18:38:36.904854 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 18:38:36.905268 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":338,"threads_started":1}
I20260927 18:38:36.907763 5736 heartbeater.cc:503] Master 127.2.95.126:37563 was elected leader, sending a full tablet report...
I20260927 18:38:36.907894 6003 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37563
I20260927 18:38:36.908193 6003 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:36.909020 6003 heartbeater.cc:511] Master 127.2.95.126:37563 requested a full tablet report, sending...
I20260927 18:38:36.910629 5547 ts_manager.cc:194] Registered new tserver with Master: ace6fe91e1004255b553d36df9dd297e (127.2.95.67:45073)
I20260927 18:38:36.911542 5547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:60951
I20260927 18:38:36.914958 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:38:36 dist-test-slave-9jll krb5kdc[5502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534316, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 18:38:36 dist-test-slave-9jll krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534316, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:36.962845 5738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:36.948408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.2.95.126:37563)
0927 18:38:36.948694 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:36.948701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:36.948966 (+ 265us) server_negotiation.cc:407] Connection header received
0927 18:38:36.949171 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:36.949176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:36.949243 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:36.949346 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:36.950731 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:36.951530 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:36.952812 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:36.953249 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:36.953365 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:36.957402 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:36.957408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:36.957411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:36.957453 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:36.960196 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:36.960957 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:36.960961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:36.960964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:36.961090 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:36.961388 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:36.961419 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:36.961421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:36.961812 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:36.961992 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:36.962415 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 18:38:36.962580 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260927 18:38:36.963528 5547 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35484
I20260927 18:38:36.965837 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 5608
Sep 27 18:38:36 dist-test-slave-9jll krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534316, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:36.981837 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:36.966911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.2.95.65:45885)
0927 18:38:36.967489 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:36.967497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:36.967525 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:38:36.967581 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:36.967589 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:36.967799 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:36.967943 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:36.969067 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:36.970200 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:36.971538 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:36.972244 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:36.972378 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:36.976211 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:36.976224 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:36.976239 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:36.976277 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:36.979017 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:36.979760 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:36.979767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:36.979770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:36.979910 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:36.980275 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:36.980335 (+ 60us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:36.980338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:36.980702 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:36.980970 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:36.981223 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 18:38:36.981487 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":287,"threads_started":1}
W20260927 18:38:36.983090 5731 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52554
W20260927 18:38:36.984230 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 5608: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:38:36.984454 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 5608
I20260927 18:38:36.997959 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 5742
Sep 27 18:38:37 dist-test-slave-9jll krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534316, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:38:37.015411 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:36.998857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.2.95.66:41929)
0927 18:38:36.999566 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:36.999573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:36.999600 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:38:36.999665 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:36.999672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:36.999877 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:37.000009 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:37.001325 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:37.002463 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:37.003907 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:37.004803 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:37.004915 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:37.009443 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:37.009461 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:37.009479 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:37.009531 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:37.012462 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:37.013313 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:37.013321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:37.013323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:37.013463 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:37.013960 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:37.013965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:37.013967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:37.014332 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:37.014571 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:37.014855 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 18:38:37.015059 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":307,"threads_started":1}
W20260927 18:38:37.016640 5865 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43002
W20260927 18:38:37.017778 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 5742: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:38:37.017930 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 5742
I20260927 18:38:37.030217 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 5875
Sep 27 18:38:37 dist-test-slave-9jll krb5kdc[5502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534316, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:38:37.048949 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:37.031292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.2.95.67:45073)
0927 18:38:37.032000 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:37.032008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:37.032048 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:38:37.032146 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:37.032155 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:37.032425 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:37.032570 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:37.033986 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:37.035095 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:37.036547 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:37.037203 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:37.037308 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:37.041867 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:37.041889 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:37.041912 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:37.041957 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:37.045649 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:37.046672 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:37.046680 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:37.046683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:37.046847 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:37.047257 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:37.047262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:37.047265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:37.047662 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:37.048020 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:37.048355 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 18:38:37.048573 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":389,"threads_started":1}
W20260927 18:38:37.050376 5998 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37402
W20260927 18:38:37.051664 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 5875: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:38:37.051836 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 5875
I20260927 18:38:37.064409 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 5515
I20260927 18:38:37.065881 5577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:37.168445 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 5515
2026-09-27T18:38:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3556 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:37 dist-test-slave-9jll krb5kdc[6020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:37 dist-test-slave-9jll krb5kdc[6020](info): set up 2 sockets
Sep 27 18:38:37 dist-test-slave-9jll krb5kdc[6020](info): commencing operation
krb5kdc: starting...
W20260927 18:38:38.174719 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:38:38 dist-test-slave-9jll krb5kdc[6020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:38:38.370191 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:42753
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:42753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:38.692303 6033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:38.692625 6033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:38.692739 6033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:38.701152 6033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:38.701297 6033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:38.701354 6033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:38.701436 6033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:38.701505 6033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:38.712018 6033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:42753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:42753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:38.713495 6033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:38.714848 6033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:38.726830 6038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:38.726862 6041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:38.726883 6039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:38.728212 6033 server_base.cc:1034] running on GCE node
I20260927 18:38:38.729069 6033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:38.730698 6033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:38.731918 6033 hybrid_clock.cc:648] HybridClock initialized: now 1790534318731857 us; error 61 us; skew 500 ppm
I20260927 18:38:38.732270 6033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:38 dist-test-slave-9jll krb5kdc[6020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534318, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:38.740589 6033 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:38:38.742342 6033 webserver.cc:460] Webserver started at http://127.2.95.126:43607/ using document root <none> and password file <none>
I20260927 18:38:38.743089 6033 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:38.743222 6033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:38.743546 6033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:38.746660 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "5d7efb72fe544996b02864249e78783c"
format_stamp: "Formatted at 2026-09-27 18:38:38 on dist-test-slave-9jll"
server_key: "763cc63fecd1818210870fa79f58223a"
server_key_iv: "a2c7c029bdb52157e0cdc8c81063f69c"
server_key_version: "encryptionkey@0"
I20260927 18:38:38.747529 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "5d7efb72fe544996b02864249e78783c"
format_stamp: "Formatted at 2026-09-27 18:38:38 on dist-test-slave-9jll"
server_key: "763cc63fecd1818210870fa79f58223a"
server_key_iv: "a2c7c029bdb52157e0cdc8c81063f69c"
server_key_version: "encryptionkey@0"
I20260927 18:38:38.753417 6033 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:38:38.757825 6048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:38.759301 6033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:38:38.759500 6033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "5d7efb72fe544996b02864249e78783c"
format_stamp: "Formatted at 2026-09-27 18:38:38 on dist-test-slave-9jll"
server_key: "763cc63fecd1818210870fa79f58223a"
server_key_iv: "a2c7c029bdb52157e0cdc8c81063f69c"
server_key_version: "encryptionkey@0"
I20260927 18:38:38.759708 6033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:38.825941 6033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:38.830416 6033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:38.830711 6033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:38.849113 6033 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:42753
I20260927 18:38:38.849148 6099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:42753 every 8 connection(s)
I20260927 18:38:38.850601 6033 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:38.854543 6100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:38.860179 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 6033
I20260927 18:38:38.860446 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:38.861068 2429 external_mini_cluster.cc:1428] Setting key 5c16ec15c6fbaba83aad258db5720810
I20260927 18:38:38.866418 6100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c: Bootstrap starting.
I20260927 18:38:38.870132 6100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:38.871351 6100 log.cc:824] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:38.874754 6100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c: No bootstrap required, opened a new log
Sep 27 18:38:38 dist-test-slave-9jll krb5kdc[6020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:38.883922 6100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7efb72fe544996b02864249e78783c" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42753 } }
I20260927 18:38:38.884441 6100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:38.884625 6100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d7efb72fe544996b02864249e78783c, State: Initialized, Role: FOLLOWER
I20260927 18:38:38.885298 6100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7efb72fe544996b02864249e78783c" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42753 } }
I20260927 18:38:38.885568 6100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:38.885725 6100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:38.885896 6100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:38.886147 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:38.863144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.2.95.126:42753)
0927 18:38:38.863985 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:38.864001 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:38.864052 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:38:38.866390 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:38.866427 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:38.866951 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:38.867420 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:38.868836 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:38.870787 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:38.872202 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:38.873013 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:38.873148 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:38.878026 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:38.878042 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:38.878059 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:38.878102 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:38.881370 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:38.882399 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:38.882406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:38.882414 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:38.882622 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:38.883034 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:38.883038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:38.883040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:38.883654 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:38.883884 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:38.884387 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 18:38:38.884851 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":334,"threads_started":1}
I20260927 18:38:38.888655 6100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7efb72fe544996b02864249e78783c" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42753 } }
I20260927 18:38:38.889273 6100 leader_election.cc:302] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d7efb72fe544996b02864249e78783c; no voters:
I20260927 18:38:38.889901 6100 leader_election.cc:288] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:38.890026 6105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:38.890654 6105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [term 1 LEADER]: Becoming Leader. State: Replica: 5d7efb72fe544996b02864249e78783c, State: Running, Role: LEADER
I20260927 18:38:38.891258 6105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7efb72fe544996b02864249e78783c" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42753 } }
I20260927 18:38:38.891848 6100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:38.894006 6106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d7efb72fe544996b02864249e78783c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7efb72fe544996b02864249e78783c" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42753 } } }
I20260927 18:38:38.894397 6106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:38.895397 6107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d7efb72fe544996b02864249e78783c. Latest consensus state: current_term: 1 leader_uuid: "5d7efb72fe544996b02864249e78783c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7efb72fe544996b02864249e78783c" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42753 } } }
I20260927 18:38:38.895715 6107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:38.897256 6113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:38.902129 6113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:38.910938 6113 catalog_manager.cc:1347] Generated new cluster ID: c2edae9eda42457c8771e454d735bf20
I20260927 18:38:38.911072 6113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:38.926985 6113 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:38.928493 6113 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:38.936949 6113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d7efb72fe544996b02864249e78783c: Generated new TSK 0
I20260927 18:38:38.938045 6113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:38:38.989742 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42753
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:39.321280 6126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:39.321571 6126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:39.321712 6126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:39.329154 6126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:39.329299 6126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:39.329457 6126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:38:39.339813 6126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:39.341302 6126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:39.342757 6126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:39.356477 6131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:39.356496 6132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:39.356448 6134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:39.357210 6126 server_base.cc:1034] running on GCE node
I20260927 18:38:39.357949 6126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:39.358953 6126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:39.360173 6126 hybrid_clock.cc:648] HybridClock initialized: now 1790534319360112 us; error 58 us; skew 500 ppm
I20260927 18:38:39.360519 6126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:39 dist-test-slave-9jll krb5kdc[6020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:39.363992 6126 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:38:39.365718 6126 webserver.cc:460] Webserver started at http://127.2.95.65:41469/ using document root <none> and password file <none>
I20260927 18:38:39.366441 6126 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:39.366564 6126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:39.366863 6126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:39.370194 6126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "843fecf8b03946a98e75c56e296a7bc1"
format_stamp: "Formatted at 2026-09-27 18:38:39 on dist-test-slave-9jll"
server_key: "fd7aa951a31b67cd17345618bdb9d620"
server_key_iv: "bef91fa61e18167faeafcc5198c71761"
server_key_version: "encryptionkey@0"
I20260927 18:38:39.371016 6126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "843fecf8b03946a98e75c56e296a7bc1"
format_stamp: "Formatted at 2026-09-27 18:38:39 on dist-test-slave-9jll"
server_key: "fd7aa951a31b67cd17345618bdb9d620"
server_key_iv: "bef91fa61e18167faeafcc5198c71761"
server_key_version: "encryptionkey@0"
I20260927 18:38:39.376539 6126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:38:39.380380 6141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:39.381925 6126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:38:39.382092 6126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "843fecf8b03946a98e75c56e296a7bc1"
format_stamp: "Formatted at 2026-09-27 18:38:39 on dist-test-slave-9jll"
server_key: "fd7aa951a31b67cd17345618bdb9d620"
server_key_iv: "bef91fa61e18167faeafcc5198c71761"
server_key_version: "encryptionkey@0"
I20260927 18:38:39.382287 6126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:39.408500 6126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:39.413012 6126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:39.413304 6126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:39.414348 6126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:39.415759 6126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:39.415839 6126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:39.415969 6126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:39.416023 6126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:39.448335 6126 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:34357
I20260927 18:38:39.448375 6253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:34357 every 8 connection(s)
I20260927 18:38:39.449764 6126 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:39.459293 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 6126
I20260927 18:38:39.459511 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:39.460196 2429 external_mini_cluster.cc:1428] Setting key d750837b89314de73d1e7c329793fc0a
Sep 27 18:38:39 dist-test-slave-9jll krb5kdc[6020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:39.472822 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:39.452753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:45005 (local address 127.2.95.126:42753)
0927 18:38:39.453340 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:39.453345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:39.454507 (+ 1162us) server_negotiation.cc:407] Connection header received
0927 18:38:39.456542 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:39.456548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:39.456641 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:39.456737 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:39.458509 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:39.459248 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:39.461098 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:39.461509 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:39.461634 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:39.466221 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:39.466227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:39.466230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:39.466268 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:39.469225 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:39.470337 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:39.470343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:39.470346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:39.470489 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:39.470967 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:39.470971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:39.470973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:39.471279 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:39.471437 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:39.472213 (+ 776us) server_negotiation.cc:299] Negotiation successful
0927 18:38:39.472472 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":351,"threads_started":1}
I20260927 18:38:39.473914 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:39.453434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:42753 (local address 127.2.95.65:45005)
0927 18:38:39.454250 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:39.454294 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:39.456207 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:39.457025 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:39.457035 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:39.457601 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:39.458291 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:39.458319 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:39.459435 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:39.459439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:39.460930 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:39.460946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:39.461782 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:39.461790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:39.462052 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:39.462950 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:39.462988 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:39.465911 (+ 2923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:39.469517 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:39.469526 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:39.469543 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:39.470171 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:39.470616 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:39.470620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:39.470622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:39.470837 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:39.471434 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:39.471444 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:39.471820 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 18:38:39.472076 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 18:38:39.472663 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":334,"threads_started":1}
I20260927 18:38:39.475651 6254 heartbeater.cc:347] Connected to a master server at 127.2.95.126:42753
I20260927 18:38:39.476011 6254 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:39.476945 6254 heartbeater.cc:511] Master 127.2.95.126:42753 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
I20260927 18:38:39.479447 6065 ts_manager.cc:194] Registered new tserver with Master: 843fecf8b03946a98e75c56e296a7bc1 (127.2.95.65:34357)
I20260927 18:38:39.481001 6065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:45005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:38:39.505965 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42753
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:39.836809 6260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:39.837112 6260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:39.837230 6260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:39.845093 6260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:39.845227 6260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:39.845383 6260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:38:39.859990 6260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:39.861546 6260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:39.863317 6260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:39.879993 6265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:39.879985 6266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:39.882454 6260 server_base.cc:1034] running on GCE node
W20260927 18:38:39.882697 6268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:39.883661 6260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:39.884831 6260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:39.888458 6260 hybrid_clock.cc:648] HybridClock initialized: now 1790534319888165 us; error 99 us; skew 500 ppm
I20260927 18:38:39.888976 6260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:39 dist-test-slave-9jll krb5kdc[6020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:39.892699 6260 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:38:39.894565 6260 webserver.cc:460] Webserver started at http://127.2.95.66:40075/ using document root <none> and password file <none>
I20260927 18:38:39.895339 6260 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:39.895455 6260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:39.895776 6260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:39.898988 6260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "e4a1082b3fd64fd89e2c34f1c3c3b7b2"
format_stamp: "Formatted at 2026-09-27 18:38:39 on dist-test-slave-9jll"
server_key: "42536dee9a9238074e99dc652f25c316"
server_key_iv: "4f58a3d3cef5d16a53aa7088a2033885"
server_key_version: "encryptionkey@0"
I20260927 18:38:39.899812 6260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "e4a1082b3fd64fd89e2c34f1c3c3b7b2"
format_stamp: "Formatted at 2026-09-27 18:38:39 on dist-test-slave-9jll"
server_key: "42536dee9a9238074e99dc652f25c316"
server_key_iv: "4f58a3d3cef5d16a53aa7088a2033885"
server_key_version: "encryptionkey@0"
I20260927 18:38:39.905774 6260 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 18:38:39.909865 6275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:39.911754 6260 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:38:39.911964 6260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "e4a1082b3fd64fd89e2c34f1c3c3b7b2"
format_stamp: "Formatted at 2026-09-27 18:38:39 on dist-test-slave-9jll"
server_key: "42536dee9a9238074e99dc652f25c316"
server_key_iv: "4f58a3d3cef5d16a53aa7088a2033885"
server_key_version: "encryptionkey@0"
I20260927 18:38:39.912176 6260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:39.933071 6260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:39.938180 6260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:39.938499 6260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:39.939792 6260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:39.941425 6260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:39.941520 6260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:39.941624 6260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:39.941711 6260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 18:38:39.974073 6260 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:42733
I20260927 18:38:39.974128 6387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:42733 every 8 connection(s)
I20260927 18:38:39.975389 6260 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:39.975965 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 6260
I20260927 18:38:39.976133 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:39.976665 2429 external_mini_cluster.cc:1428] Setting key 687947c4b0b8122d64b3f64f050fe93c
Sep 27 18:38:39 dist-test-slave-9jll krb5kdc[6020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
I20260927 18:38:39.997504 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:39.977938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:40653 (local address 127.2.95.126:42753)
0927 18:38:39.978340 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:39.978344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:39.979612 (+ 1268us) server_negotiation.cc:407] Connection header received
0927 18:38:39.981490 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:39.981496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:39.981584 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:39.981744 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:39.983414 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:39.984194 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:39.986569 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:39.987024 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:39.987203 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:39.991146 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:39.991152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:39.991155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:39.991190 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:39.993933 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:39.994742 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:39.994746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:39.994749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:39.994987 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:39.995478 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:39.995482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:39.995484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:39.995925 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:39.996079 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:39.996863 (+ 784us) server_negotiation.cc:299] Negotiation successful
0927 18:38:39.997098 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":212,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
I20260927 18:38:39.998560 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:39.978577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:42753 (local address 127.2.95.66:40653)
0927 18:38:39.979414 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:39.979467 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:39.981179 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:39.982052 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:39.982061 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:39.982602 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:39.983214 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:39.983233 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:39.984365 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:39.984371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:39.986401 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:39.986419 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:39.987260 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:39.987266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:39.987542 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:39.988275 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:39.988308 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:39.990884 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:39.994097 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:39.994107 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:39.994125 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:39.994589 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:39.995120 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:39.995124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:39.995139 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:39.995344 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:39.996077 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:39.996085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:39.996476 (+ 391us) client_negotiation.cc:769] Sending connection context
0927 18:38:39.996761 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 18:38:39.997317 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":350,"threads_started":1}
I20260927 18:38:40.000448 6388 heartbeater.cc:347] Connected to a master server at 127.2.95.126:42753
I20260927 18:38:40.000881 6388 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:40.001993 6388 heartbeater.cc:511] Master 127.2.95.126:42753 requested a full tablet report, sending...
I20260927 18:38:40.003775 6065 ts_manager.cc:194] Registered new tserver with Master: e4a1082b3fd64fd89e2c34f1c3c3b7b2 (127.2.95.66:42733)
I20260927 18:38:40.004632 6065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:40653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:38:40.025662 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42753
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:40.358999 6394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:40.359299 6394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:40.359427 6394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:40.366954 6394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:40.367085 6394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:40.367230 6394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:38:40.377167 6394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:40.378533 6394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:40.379825 6394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:40.392966 6402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:40.392877 6400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:40.392941 6399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:40.394553 6394 server_base.cc:1034] running on GCE node
I20260927 18:38:40.395272 6394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:40.396370 6394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:40.397586 6394 hybrid_clock.cc:648] HybridClock initialized: now 1790534320397515 us; error 63 us; skew 500 ppm
I20260927 18:38:40.397943 6394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:40 dist-test-slave-9jll krb5kdc[6020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:40.401245 6394 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:38:40.402837 6394 webserver.cc:460] Webserver started at http://127.2.95.67:37767/ using document root <none> and password file <none>
I20260927 18:38:40.403527 6394 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:40.403644 6394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:40.403937 6394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:40.406991 6394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "1815567a6e274f44874c1c450ec5316f"
format_stamp: "Formatted at 2026-09-27 18:38:40 on dist-test-slave-9jll"
server_key: "7858ed3f021b814c6b5509ac4b19d8e0"
server_key_iv: "bc66716dd773207843275ea6dcc37ca8"
server_key_version: "encryptionkey@0"
I20260927 18:38:40.407804 6394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "1815567a6e274f44874c1c450ec5316f"
format_stamp: "Formatted at 2026-09-27 18:38:40 on dist-test-slave-9jll"
server_key: "7858ed3f021b814c6b5509ac4b19d8e0"
server_key_iv: "bc66716dd773207843275ea6dcc37ca8"
server_key_version: "encryptionkey@0"
I20260927 18:38:40.413450 6394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 18:38:40.417492 6409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:40.419067 6394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:38:40.419243 6394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "1815567a6e274f44874c1c450ec5316f"
format_stamp: "Formatted at 2026-09-27 18:38:40 on dist-test-slave-9jll"
server_key: "7858ed3f021b814c6b5509ac4b19d8e0"
server_key_iv: "bc66716dd773207843275ea6dcc37ca8"
server_key_version: "encryptionkey@0"
I20260927 18:38:40.419466 6394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:40.433821 6394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:40.438351 6394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:40.438668 6394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:40.439791 6394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:40.441262 6394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:40.441341 6394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:40.441460 6394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:40.441516 6394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:40.473592 6394 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:42163
I20260927 18:38:40.473726 6521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:42163 every 8 connection(s)
I20260927 18:38:40.474874 6394 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:38:40.484241 6254 heartbeater.cc:503] Master 127.2.95.126:42753 was elected leader, sending a full tablet report...
I20260927 18:38:40.485179 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 6394
I20260927 18:38:40.485378 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:40.485944 2429 external_mini_cluster.cc:1428] Setting key 5272c7152831ab66417f23866133f2ca
Sep 27 18:38:40 dist-test-slave-9jll krb5kdc[6020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:40.495782 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:40.477445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:44181 (local address 127.2.95.126:42753)
0927 18:38:40.477639 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:40.477644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:40.478770 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 18:38:40.480608 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:40.480614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:40.480707 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:40.480795 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:40.482362 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.483105 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.484703 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.485258 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.485396 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:40.489614 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:40.489618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:40.489622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:40.489656 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:40.492530 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:40.493395 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:40.493400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:40.493403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:40.493533 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:40.494091 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:40.494094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:40.494097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:40.494388 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:40.494512 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:40.495320 (+ 808us) server_negotiation.cc:299] Negotiation successful
0927 18:38:40.495537 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:38:40.496869 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:40.477923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:42753 (local address 127.2.95.67:44181)
0927 18:38:40.478616 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:40.478661 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:40.480335 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:40.481042 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:40.481050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:40.481555 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:40.482187 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:40.482206 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.483255 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.483259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:40.484569 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:40.484581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.485601 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.485610 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:40.485983 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:40.486751 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:40.486783 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:40.489385 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:40.492701 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:40.492710 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:40.492724 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:40.493207 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:40.493636 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:40.493641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:40.493644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:40.493949 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:40.494515 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:40.494522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:40.494942 (+ 420us) client_negotiation.cc:769] Sending connection context
0927 18:38:40.495214 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 18:38:40.495716 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":266,"threads_started":1}
I20260927 18:38:40.498473 6522 heartbeater.cc:347] Connected to a master server at 127.2.95.126:42753
I20260927 18:38:40.498783 6522 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:40.499595 6522 heartbeater.cc:511] Master 127.2.95.126:42753 requested a full tablet report, sending...
I20260927 18:38:40.501035 6065 ts_manager.cc:194] Registered new tserver with Master: 1815567a6e274f44874c1c450ec5316f (127.2.95.67:42163)
I20260927 18:38:40.501628 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:38:40.501929 6065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:44181
Sep 27 18:38:40 dist-test-slave-9jll krb5kdc[6020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:38:40 dist-test-slave-9jll krb5kdc[6020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:40.552037 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:40.535709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.2.95.65:34357)
0927 18:38:40.536492 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:40.536501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:40.536529 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:38:40.536589 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:40.536596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:40.536815 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:40.536980 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:40.538597 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.539660 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.541010 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.541717 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.541862 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:40.545993 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:40.546007 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:40.546023 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:40.546074 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:40.548932 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:40.549787 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:40.549795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:40.549798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:40.549988 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:40.550451 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:40.550456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:40.550458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:40.550931 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:40.551180 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:40.551501 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 18:38:40.551729 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":383,"threads_started":1}
Sep 27 18:38:40 dist-test-slave-9jll krb5kdc[6020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:40.570745 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:40.556634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.2.95.126:42753)
0927 18:38:40.556843 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:40.556849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:40.557165 (+ 316us) server_negotiation.cc:407] Connection header received
0927 18:38:40.557340 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:40.557345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:40.557421 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:40.557503 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:40.558873 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.559667 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.560933 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.561351 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.561462 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:40.565547 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:40.565552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:40.565555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:40.565594 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:40.568124 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:40.568817 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:40.568824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:40.568826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:40.568958 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:40.569295 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:40.569299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:40.569301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:40.569623 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:40.569909 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:40.570210 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 18:38:40.570428 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 18:38:40.571293 6065 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55134
I20260927 18:38:40.572434 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 6126
I20260927 18:38:40.585368 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:40.573439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.2.95.65:34357)
0927 18:38:40.573649 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:40.573655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:40.573961 (+ 306us) server_negotiation.cc:407] Connection header received
0927 18:38:40.574117 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:40.574120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:40.574219 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:40.574312 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:40.575576 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.576248 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.577436 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.577888 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.578006 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:40.580302 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:40.580307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:40.580310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:40.580351 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:40.582903 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:40.583566 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:40.583572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:40.583574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:40.583702 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:40.584052 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:40.584056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:40.584058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:40.584315 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:40.584533 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:40.584848 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 18:38:40.585064 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 18:38:40.586007 6249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:40.668783 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 6126
I20260927 18:38:40.682114 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 6260
Sep 27 18:38:40 dist-test-slave-9jll krb5kdc[6020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:38:40.698311 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:40.683067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.2.95.66:42733)
0927 18:38:40.683613 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:40.683621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:40.683647 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:38:40.683705 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:40.683712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:40.684017 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:40.684157 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:40.685374 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.686590 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.687905 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.688523 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.688640 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:40.692719 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:40.692734 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:40.692750 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:40.692795 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:40.695745 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:40.696522 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:40.696530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:40.696532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:40.696656 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:40.696954 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:40.696958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:40.696960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:40.697305 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:40.697532 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:40.697836 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 18:38:40.698035 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":284,"threads_started":1}
I20260927 18:38:40.699623 6383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:40.784638 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 6260
I20260927 18:38:40.797957 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 6394
Sep 27 18:38:40 dist-test-slave-9jll krb5kdc[6020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:38:40.815758 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:40.798927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.2.95.67:42163)
0927 18:38:40.799574 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:40.799582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:40.799613 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:38:40.799671 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:40.799678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:40.799905 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:40.800046 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:40.801403 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.802608 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.804127 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:40.804742 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:40.804872 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:40.809663 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:40.809717 (+ 54us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:40.809753 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:40.809802 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:40.812602 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:40.813458 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:40.813466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:40.813469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:40.813614 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:40.814055 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:40.814059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:40.814061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:40.814473 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:40.814703 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:40.815143 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 18:38:40.815365 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":335,"threads_started":1}
I20260927 18:38:40.817260 6517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:40.901134 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 6394
I20260927 18:38:40.914233 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 6033
I20260927 18:38:40.915416 6095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:41.019781 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 6033
2026-09-27T18:38:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3852 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:41 dist-test-slave-9jll krb5kdc[6541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:41 dist-test-slave-9jll krb5kdc[6541](info): set up 2 sockets
Sep 27 18:38:41 dist-test-slave-9jll krb5kdc[6541](info): commencing operation
krb5kdc: starting...
W20260927 18:38:42.034523 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:38:42 dist-test-slave-9jll krb5kdc[6541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:38:42.243234 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:40399
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:40399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:42.591672 6554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:42.592010 6554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:42.592137 6554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:42.599602 6554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:42.599745 6554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:42.599830 6554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:42.599903 6554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:42.599974 6554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:42.610033 6554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:40399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:40399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:42.611368 6554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:42.612660 6554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:42.623879 6562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:42.624009 6559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:42.624027 6560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:42.624552 6554 server_base.cc:1034] running on GCE node
I20260927 18:38:42.625329 6554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:42.626829 6554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:42.628055 6554 hybrid_clock.cc:648] HybridClock initialized: now 1790534322628001 us; error 43 us; skew 500 ppm
I20260927 18:38:42.628414 6554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:42 dist-test-slave-9jll krb5kdc[6541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534322, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:42.631798 6554 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:38:42.633132 6554 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:42.633242 6554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:42.633561 6554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:42.636620 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "cf6e7bb4c56c4c358602cce9473f57b8"
format_stamp: "Formatted at 2026-09-27 18:38:42 on dist-test-slave-9jll"
server_key: "9c7d9dde9ccd73ddaac7d90b1306435a"
server_key_iv: "dd37a63151353625ece4444d99650da1"
server_key_version: "encryptionkey@0"
I20260927 18:38:42.637423 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "cf6e7bb4c56c4c358602cce9473f57b8"
format_stamp: "Formatted at 2026-09-27 18:38:42 on dist-test-slave-9jll"
server_key: "9c7d9dde9ccd73ddaac7d90b1306435a"
server_key_iv: "dd37a63151353625ece4444d99650da1"
server_key_version: "encryptionkey@0"
I20260927 18:38:42.642656 6554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:38:42.646543 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:42.648046 6554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:38:42.648216 6554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "cf6e7bb4c56c4c358602cce9473f57b8"
format_stamp: "Formatted at 2026-09-27 18:38:42 on dist-test-slave-9jll"
server_key: "9c7d9dde9ccd73ddaac7d90b1306435a"
server_key_iv: "dd37a63151353625ece4444d99650da1"
server_key_version: "encryptionkey@0"
I20260927 18:38:42.648417 6554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:42.669111 6554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:42.673841 6554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:42.674106 6554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:42.691887 6554 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:40399
I20260927 18:38:42.691906 6619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:40399 every 8 connection(s)
I20260927 18:38:42.692790 6554 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:42.696575 6620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:42.703289 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 6554
I20260927 18:38:42.703547 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:42.704331 2429 external_mini_cluster.cc:1428] Setting key b657b7f4b6e759f780edf321392c6970
I20260927 18:38:42.708596 6620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8: Bootstrap starting.
I20260927 18:38:42.712527 6620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:42.713944 6620 log.cc:824] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:42.717116 6620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8: No bootstrap required, opened a new log
Sep 27 18:38:42 dist-test-slave-9jll krb5kdc[6541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:42.726641 6620 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6e7bb4c56c4c358602cce9473f57b8" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40399 } }
I20260927 18:38:42.727130 6620 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:42.727239 6620 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6e7bb4c56c4c358602cce9473f57b8, State: Initialized, Role: FOLLOWER
I20260927 18:38:42.727941 6620 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6e7bb4c56c4c358602cce9473f57b8" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40399 } }
I20260927 18:38:42.728214 6620 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:42.728366 6620 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:42.728545 6620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:42.731051 6623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:42.706610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.2.95.126:40399)
0927 18:38:42.707426 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:42.707442 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:42.707504 (+ 62us) server_negotiation.cc:407] Connection header received
0927 18:38:42.709911 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:42.709941 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:42.710578 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:42.711112 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:42.712594 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:42.714651 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:42.716195 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:42.717213 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:42.717386 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:42.721409 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:42.721421 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:42.721447 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:42.721554 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:42.725970 (+ 4416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:42.726834 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:42.726843 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:42.726852 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:42.727072 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:42.727520 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:42.727525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:42.727528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:42.728408 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:42.728700 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:42.729097 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 18:38:42.729582 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":355,"threads_started":1}
I20260927 18:38:42.731870 6620 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6e7bb4c56c4c358602cce9473f57b8" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40399 } }
I20260927 18:38:42.732440 6620 leader_election.cc:302] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf6e7bb4c56c4c358602cce9473f57b8; no voters:
I20260927 18:38:42.733108 6620 leader_election.cc:288] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:42.733248 6625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:42.733924 6625 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [term 1 LEADER]: Becoming Leader. State: Replica: cf6e7bb4c56c4c358602cce9473f57b8, State: Running, Role: LEADER
I20260927 18:38:42.734516 6625 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6e7bb4c56c4c358602cce9473f57b8" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40399 } }
I20260927 18:38:42.735996 6620 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:42.737104 6626 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf6e7bb4c56c4c358602cce9473f57b8. Latest consensus state: current_term: 1 leader_uuid: "cf6e7bb4c56c4c358602cce9473f57b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6e7bb4c56c4c358602cce9473f57b8" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40399 } } }
I20260927 18:38:42.737423 6626 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:42.738262 6627 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf6e7bb4c56c4c358602cce9473f57b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6e7bb4c56c4c358602cce9473f57b8" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40399 } } }
I20260927 18:38:42.738677 6627 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:42.743466 6630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:42.749030 6630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:42.759259 6630 catalog_manager.cc:1347] Generated new cluster ID: 26eeb34d54c1441f96a2580c3b1648b8
I20260927 18:38:42.759378 6630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:42.772555 6630 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:42.774020 6630 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:42.782230 6630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8: Generated new TSK 0
I20260927 18:38:42.783336 6630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:38:42.834579 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:40399
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:43.167564 6646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:43.167867 6646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:43.167991 6646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:43.175372 6646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:43.175500 6646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:43.175647 6646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:38:43.185557 6646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:40399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:43.186895 6646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:43.188212 6646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:43.201330 6651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:43.201359 6652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:43.201330 6654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:43.202342 6646 server_base.cc:1034] running on GCE node
I20260927 18:38:43.203044 6646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:43.204038 6646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:43.205260 6646 hybrid_clock.cc:648] HybridClock initialized: now 1790534323205186 us; error 66 us; skew 500 ppm
I20260927 18:38:43.205617 6646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:43 dist-test-slave-9jll krb5kdc[6541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534323, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:43.209004 6646 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:38:43.210363 6646 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:43.210474 6646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:43.210790 6646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:43.213948 6646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "1b190d56ff344f4eb2515909df76101b"
format_stamp: "Formatted at 2026-09-27 18:38:43 on dist-test-slave-9jll"
server_key: "dbd590669d4dc4b53b9eaeca42ab40d5"
server_key_iv: "763baef0274ec3281711e0200db4c36a"
server_key_version: "encryptionkey@0"
I20260927 18:38:43.214793 6646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "1b190d56ff344f4eb2515909df76101b"
format_stamp: "Formatted at 2026-09-27 18:38:43 on dist-test-slave-9jll"
server_key: "dbd590669d4dc4b53b9eaeca42ab40d5"
server_key_iv: "763baef0274ec3281711e0200db4c36a"
server_key_version: "encryptionkey@0"
I20260927 18:38:43.220238 6646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:38:43.224154 6660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:43.225601 6646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:38:43.225821 6646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "1b190d56ff344f4eb2515909df76101b"
format_stamp: "Formatted at 2026-09-27 18:38:43 on dist-test-slave-9jll"
server_key: "dbd590669d4dc4b53b9eaeca42ab40d5"
server_key_iv: "763baef0274ec3281711e0200db4c36a"
server_key_version: "encryptionkey@0"
I20260927 18:38:43.226045 6646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:43.245172 6646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:43.249781 6646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:43.250047 6646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:43.251039 6646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:43.252511 6646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:43.252599 6646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:43.252707 6646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:43.252784 6646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:43.283614 6646 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:42275
I20260927 18:38:43.283653 6772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:42275 every 8 connection(s)
I20260927 18:38:43.284456 6646 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:43.294610 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 6646
I20260927 18:38:43.294845 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:43.295471 2429 external_mini_cluster.cc:1428] Setting key f1ffba4cb767ee9f11b484e068816aff
Sep 27 18:38:43 dist-test-slave-9jll krb5kdc[6541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534323, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:43.305508 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:43.286993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:51169 (local address 127.2.95.126:40399)
0927 18:38:43.287449 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:43.287453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:43.288518 (+ 1065us) server_negotiation.cc:407] Connection header received
0927 18:38:43.290334 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:43.290338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:43.290439 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:43.290540 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:43.292099 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:43.292817 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:43.294950 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:43.295372 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:43.295481 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:43.299310 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:43.299325 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:43.299329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:43.299369 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:43.302377 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:43.303224 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:43.303228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:43.303231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:43.303357 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:43.303827 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:43.303830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:43.303832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:43.304077 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:43.304188 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:43.305061 (+ 873us) server_negotiation.cc:299] Negotiation successful
0927 18:38:43.305242 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
I20260927 18:38:43.307006 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:43.287616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:40399 (local address 127.2.95.65:51169)
0927 18:38:43.288340 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:43.288391 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:43.290056 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:43.290725 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:43.290734 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:43.291338 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:43.291921 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:43.291943 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:43.293013 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:43.293017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:43.294704 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:43.294721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:43.295630 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:43.295638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:43.295947 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:43.296718 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:43.296742 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:43.299057 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:43.302569 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:43.302579 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:43.302600 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:43.303082 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:43.303462 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:43.303467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:43.303471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:43.303732 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:43.304195 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:43.304204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:43.304669 (+ 465us) client_negotiation.cc:769] Sending connection context
0927 18:38:43.304961 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 18:38:43.305623 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":313,"threads_started":1}
I20260927 18:38:43.308696 6773 heartbeater.cc:347] Connected to a master server at 127.2.95.126:40399
I20260927 18:38:43.308996 6773 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:43.309660 6773 heartbeater.cc:511] Master 127.2.95.126:40399 requested a full tablet report, sending...
I20260927 18:38:43.311811 6585 ts_manager.cc:194] Registered new tserver with Master: 1b190d56ff344f4eb2515909df76101b (127.2.95.65:42275)
I20260927 18:38:43.313273 6585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:51169
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:38:43.342231 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:40399
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:43.668171 6779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:43.668454 6779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:43.668576 6779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:43.675836 6779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:43.675966 6779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:43.676113 6779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:38:43.686309 6779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:40399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:43.687659 6779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:43.688941 6779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:43.703277 6785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:43.703387 6784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:43.703330 6787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:43.704010 6779 server_base.cc:1034] running on GCE node
I20260927 18:38:43.704669 6779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:43.705662 6779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:43.707020 6779 hybrid_clock.cc:648] HybridClock initialized: now 1790534323706937 us; error 74 us; skew 500 ppm
I20260927 18:38:43.707414 6779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:43 dist-test-slave-9jll krb5kdc[6541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534323, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:43.710892 6779 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:38:43.712121 6779 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:43.712231 6779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:43.712553 6779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:43.715760 6779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "8ff2ac78883c4886ba4f761ff2d8a6af"
format_stamp: "Formatted at 2026-09-27 18:38:43 on dist-test-slave-9jll"
server_key: "85ebf0960977b333d10c0fd2f04915b2"
server_key_iv: "09424d611820daefe0a92a0718c9a7b1"
server_key_version: "encryptionkey@0"
I20260927 18:38:43.716625 6779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "8ff2ac78883c4886ba4f761ff2d8a6af"
format_stamp: "Formatted at 2026-09-27 18:38:43 on dist-test-slave-9jll"
server_key: "85ebf0960977b333d10c0fd2f04915b2"
server_key_iv: "09424d611820daefe0a92a0718c9a7b1"
server_key_version: "encryptionkey@0"
I20260927 18:38:43.722358 6779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:38:43.726362 6793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:43.727955 6779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:38:43.728127 6779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "8ff2ac78883c4886ba4f761ff2d8a6af"
format_stamp: "Formatted at 2026-09-27 18:38:43 on dist-test-slave-9jll"
server_key: "85ebf0960977b333d10c0fd2f04915b2"
server_key_iv: "09424d611820daefe0a92a0718c9a7b1"
server_key_version: "encryptionkey@0"
I20260927 18:38:43.728346 6779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:43.752806 6779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:43.758055 6779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:43.758340 6779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:43.759334 6779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:43.760747 6779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:43.760854 6779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:43.760972 6779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:43.761039 6779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:43.792480 6779 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:40271
I20260927 18:38:43.792536 6905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:40271 every 8 connection(s)
I20260927 18:38:43.793365 6779 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:43.802371 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 6779
I20260927 18:38:43.802552 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:43.803053 2429 external_mini_cluster.cc:1428] Setting key afc1dabc235d9919fb2625f8da633f98
Sep 27 18:38:43 dist-test-slave-9jll krb5kdc[6541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534323, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:43.814462 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:43.795827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:38945 (local address 127.2.95.126:40399)
0927 18:38:43.796088 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:43.796093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:43.797226 (+ 1133us) server_negotiation.cc:407] Connection header received
0927 18:38:43.799090 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:43.799095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:43.799184 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:43.799269 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:43.800891 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:43.801609 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:43.803464 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:43.803988 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:43.804108 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:43.808279 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:43.808284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:43.808288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:43.808325 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:43.811006 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:43.811845 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:43.811851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:43.811854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:43.812089 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:43.812511 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:43.812515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:43.812518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:43.812860 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:43.813007 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:43.813918 (+ 911us) server_negotiation.cc:299] Negotiation successful
0927 18:38:43.814150 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:38:43.815431 6908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:43.796302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:40399 (local address 127.2.95.66:38945)
0927 18:38:43.797055 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:43.797095 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:43.798788 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:43.799526 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:43.799534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:43.800100 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:43.800699 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:43.800717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:43.801788 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:43.801792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:43.803305 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:43.803319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:43.804255 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:43.804264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:43.804634 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:43.805522 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:43.805547 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:43.808021 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:43.811186 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:43.811194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:43.811210 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:43.811718 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:43.812202 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:43.812206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:43.812208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:43.812412 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:43.813077 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:43.813084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:43.813449 (+ 365us) client_negotiation.cc:769] Sending connection context
0927 18:38:43.813837 (+ 388us) client_negotiation.cc:240] Negotiation successful
0927 18:38:43.814269 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":308,"threads_started":1}
I20260927 18:38:43.817204 6906 heartbeater.cc:347] Connected to a master server at 127.2.95.126:40399
I20260927 18:38:43.817564 6906 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:43.818372 6906 heartbeater.cc:511] Master 127.2.95.126:40399 requested a full tablet report, sending...
I20260927 18:38:43.819977 6585 ts_manager.cc:194] Registered new tserver with Master: 8ff2ac78883c4886ba4f761ff2d8a6af (127.2.95.66:40271)
I20260927 18:38:43.821002 6585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:38945
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:38:43.850096 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:40399
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:44.180047 6911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:44.180335 6911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:44.180430 6911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:44.187891 6911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:44.188032 6911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:44.188184 6911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:38:44.198330 6911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:40399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:44.199620 6911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:44.200894 6911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:44.213915 6917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:44.213964 6916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:44.214600 6919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:44.214872 6911 server_base.cc:1034] running on GCE node
I20260927 18:38:44.215497 6911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:44.216527 6911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:44.217769 6911 hybrid_clock.cc:648] HybridClock initialized: now 1790534324217712 us; error 50 us; skew 500 ppm
I20260927 18:38:44.218151 6911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:44 dist-test-slave-9jll krb5kdc[6541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534324, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:44.221454 6911 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:38:44.222929 6911 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:44.223037 6911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:44.223385 6911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:44.226656 6911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "810a404a0f1d4d6991116bdddec2ea3d"
format_stamp: "Formatted at 2026-09-27 18:38:44 on dist-test-slave-9jll"
server_key: "836aeb69f1a7afcf6afb969edc15e74d"
server_key_iv: "f40c35c5186897fcd5750496df7f751c"
server_key_version: "encryptionkey@0"
I20260927 18:38:44.227478 6911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "810a404a0f1d4d6991116bdddec2ea3d"
format_stamp: "Formatted at 2026-09-27 18:38:44 on dist-test-slave-9jll"
server_key: "836aeb69f1a7afcf6afb969edc15e74d"
server_key_iv: "f40c35c5186897fcd5750496df7f751c"
server_key_version: "encryptionkey@0"
I20260927 18:38:44.232981 6911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:38:44.236872 6925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:44.238535 6911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:38:44.238693 6911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "810a404a0f1d4d6991116bdddec2ea3d"
format_stamp: "Formatted at 2026-09-27 18:38:44 on dist-test-slave-9jll"
server_key: "836aeb69f1a7afcf6afb969edc15e74d"
server_key_iv: "f40c35c5186897fcd5750496df7f751c"
server_key_version: "encryptionkey@0"
I20260927 18:38:44.238950 6911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:44.252130 6911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:44.256518 6911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:44.256877 6911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:44.258025 6911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:44.259454 6911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:44.259548 6911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:44.259668 6911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:44.259740 6911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:44.291256 6911 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:40359
I20260927 18:38:44.291301 7037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:40359 every 8 connection(s)
I20260927 18:38:44.292095 6911 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:38:44.300328 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 6911
I20260927 18:38:44.300514 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:44.300905 2429 external_mini_cluster.cc:1428] Setting key a940c143db8d85e540d1bcb4f63fcd67
Sep 27 18:38:44 dist-test-slave-9jll krb5kdc[6541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534324, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:44.312954 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.294641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:41753 (local address 127.2.95.126:40399)
0927 18:38:44.294964 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:44.294969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:44.296027 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 18:38:44.297953 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:44.297958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:44.298045 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:44.298138 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:44.299817 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.300677 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.302650 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.303188 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.303288 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.306960 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:44.306967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:44.306972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:44.307008 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:44.309708 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:44.310625 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:44.310629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:44.310631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:44.310767 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:44.311273 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:44.311277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:44.311279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:44.311550 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:44.311662 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:44.312526 (+ 864us) server_negotiation.cc:299] Negotiation successful
0927 18:38:44.312698 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 18:38:44.313979 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.295121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:40399 (local address 127.2.95.67:41753)
0927 18:38:44.295855 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:44.295901 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:44.297619 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:44.298453 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:44.298462 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:44.299014 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:44.299611 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:44.299629 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.300843 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.300847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:44.302489 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:44.302503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.303448 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.303457 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:44.303765 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.304420 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:44.304449 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:44.306713 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:44.309967 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:44.309977 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:44.309993 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:44.310483 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:44.310955 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:44.310959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:44.310961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:44.311163 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:44.311669 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:44.311676 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:44.312129 (+ 453us) client_negotiation.cc:769] Sending connection context
0927 18:38:44.312375 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 18:38:44.312786 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":296,"threads_started":1}
I20260927 18:38:44.315516 7038 heartbeater.cc:347] Connected to a master server at 127.2.95.126:40399
I20260927 18:38:44.315825 7038 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:44.316555 7038 heartbeater.cc:511] Master 127.2.95.126:40399 requested a full tablet report, sending...
I20260927 18:38:44.316808 6773 heartbeater.cc:503] Master 127.2.95.126:40399 was elected leader, sending a full tablet report...
I20260927 18:38:44.318106 6585 ts_manager.cc:194] Registered new tserver with Master: 810a404a0f1d4d6991116bdddec2ea3d (127.2.95.67:40359)
I20260927 18:38:44.318964 6585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:41753
I20260927 18:38:44.331439 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:38:44.353765 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.337982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.2.95.126:40399)
0927 18:38:44.338211 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:44.338218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:44.338685 (+ 467us) server_negotiation.cc:407] Connection header received
0927 18:38:44.338887 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:44.338891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:44.338985 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:44.339088 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:44.341001 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.342156 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.344213 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.344934 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.345072 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.348093 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:44.348101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:44.348106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:44.348169 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:44.350822 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:44.351576 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:44.351582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:44.351585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:44.351752 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:44.352309 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:44.352314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:44.352318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:44.352670 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:44.352928 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:44.353296 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 18:38:44.353472 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:38:44.358008 6585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:38:44.365345 6585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:44.411520 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.379494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.2.95.67:40359)
0927 18:38:44.402766 (+ 23272us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:44.402774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:44.403456 (+ 682us) server_negotiation.cc:407] Connection header received
0927 18:38:44.403556 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:44.403563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:44.403779 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:44.403967 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:44.405065 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.406177 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.409217 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.410317 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.410449 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.410697 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:44.410985 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 18:38:44.411170 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23117,"thread_start_us":22954,"threads_started":1}
I20260927 18:38:44.411563 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.379108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:40359 (local address 127.0.0.1:37534)
0927 18:38:44.403179 (+ 24071us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:44.403193 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:44.403306 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:44.404210 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:44.404219 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:44.404255 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:44.404883 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:44.404892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.406344 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.406349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:44.409053 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:44.409067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.410544 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.410551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:44.410751 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.410794 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 18:38:44.410929 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 18:38:44.411147 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23964,"mutex_wait_us":766,"spinlock_wait_cycles":24192,"thread_start_us":23853,"threads_started":1}
I20260927 18:38:44.412683 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.378681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:40271 (local address 127.0.0.1:46096)
0927 18:38:44.401404 (+ 22723us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:44.401472 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:44.402834 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:44.404119 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:44.404126 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:44.404155 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:44.404531 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:44.404540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.406315 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.406321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:44.408709 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:44.408727 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.410916 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.410923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:44.411099 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.411133 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:38:44.412102 (+ 969us) client_negotiation.cc:240] Negotiation successful
0927 18:38:44.412242 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22381,"mutex_wait_us":89,"thread_start_us":22204,"threads_started":1}
I20260927 18:38:44.413380 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.378655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.2.95.66:40271)
0927 18:38:44.397352 (+ 18697us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:44.397365 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:44.401656 (+ 4291us) server_negotiation.cc:407] Connection header received
0927 18:38:44.403351 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:44.403360 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:44.403628 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:44.403814 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:44.404689 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.406170 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.408889 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.410464 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.410575 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.410760 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:44.412929 (+ 2169us) server_negotiation.cc:299] Negotiation successful
0927 18:38:44.413105 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18478,"thread_start_us":18338,"threads_started":1}
I20260927 18:38:44.415697 6973 tablet_service.cc:1467] Processing CreateTablet for tablet bdb20fcf28d14fdf9d6bb6160b9c039b (DEFAULT_TABLE table=test-table [id=f45b0b3f383d4d5a8f3c79fe62b692df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:44.416534 6841 tablet_service.cc:1467] Processing CreateTablet for tablet bdb20fcf28d14fdf9d6bb6160b9c039b (DEFAULT_TABLE table=test-table [id=f45b0b3f383d4d5a8f3c79fe62b692df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:44.417156 6973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb20fcf28d14fdf9d6bb6160b9c039b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:44.417845 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.380063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.2.95.65:42275)
0927 18:38:44.402264 (+ 22201us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:44.402274 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:44.403459 (+ 1185us) server_negotiation.cc:407] Connection header received
0927 18:38:44.403556 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:44.403563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:44.403786 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:44.403963 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:44.405021 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.406516 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.410638 (+ 4122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.412116 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.412215 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.412427 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:44.417309 (+ 4882us) server_negotiation.cc:299] Negotiation successful
0927 18:38:44.417508 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22013,"thread_start_us":21853,"threads_started":1}
I20260927 18:38:44.418009 6841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb20fcf28d14fdf9d6bb6160b9c039b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:44.417038 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.379033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:42275 (local address 127.0.0.1:58930)
0927 18:38:44.403126 (+ 24093us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:44.403147 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:44.403306 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:44.404210 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:44.404215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:44.404258 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:44.404809 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:44.404825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.406698 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.406703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:44.410418 (+ 3715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:44.410434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.412616 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.412626 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:44.412805 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.412835 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:38:44.416444 (+ 3609us) client_negotiation.cc:240] Negotiation successful
0927 18:38:44.416610 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23916,"mutex_wait_us":1493,"spinlock_wait_cycles":10368,"thread_start_us":23998,"threads_started":1}
I20260927 18:38:44.422415 6708 tablet_service.cc:1467] Processing CreateTablet for tablet bdb20fcf28d14fdf9d6bb6160b9c039b (DEFAULT_TABLE table=test-table [id=f45b0b3f383d4d5a8f3c79fe62b692df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:44.424315 6708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb20fcf28d14fdf9d6bb6160b9c039b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:44.429955 7056 tablet_bootstrap.cc:492] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af: Bootstrap starting.
I20260927 18:38:44.434532 7057 tablet_bootstrap.cc:492] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d: Bootstrap starting.
I20260927 18:38:44.434859 7056 tablet_bootstrap.cc:654] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:44.436215 7056 log.cc:824] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:44.438622 7057 tablet_bootstrap.cc:654] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:44.439744 7056 tablet_bootstrap.cc:492] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af: No bootstrap required, opened a new log
I20260927 18:38:44.440128 7056 ts_tablet_manager.cc:1397] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 18:38:44.440514 7057 log.cc:824] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:44.440873 7059 tablet_bootstrap.cc:492] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b: Bootstrap starting.
I20260927 18:38:44.444088 7057 tablet_bootstrap.cc:492] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d: No bootstrap required, opened a new log
I20260927 18:38:44.444396 7057 ts_tablet_manager.cc:1397] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 18:38:44.444518 7059 tablet_bootstrap.cc:654] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:44.445914 7059 log.cc:824] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:44.448341 7056 raft_consensus.cc:348] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40271 } } peers { permanent_uuid: "1b190d56ff344f4eb2515909df76101b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42275 } } peers { permanent_uuid: "810a404a0f1d4d6991116bdddec2ea3d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40359 } }
I20260927 18:38:44.448720 7056 raft_consensus.cc:374] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:44.448840 7056 raft_consensus.cc:729] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff2ac78883c4886ba4f761ff2d8a6af, State: Initialized, Role: FOLLOWER
I20260927 18:38:44.449527 7059 tablet_bootstrap.cc:492] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b: No bootstrap required, opened a new log
I20260927 18:38:44.449424 7056 consensus_queue.cc:260] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40271 } } peers { permanent_uuid: "1b190d56ff344f4eb2515909df76101b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42275 } } peers { permanent_uuid: "810a404a0f1d4d6991116bdddec2ea3d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40359 } }
I20260927 18:38:44.449889 7059 ts_tablet_manager.cc:1397] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260927 18:38:44.450634 6906 heartbeater.cc:503] Master 127.2.95.126:40399 was elected leader, sending a full tablet report...
I20260927 18:38:44.451386 7056 ts_tablet_manager.cc:1428] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 18:38:44.454664 7057 raft_consensus.cc:348] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40271 } } peers { permanent_uuid: "1b190d56ff344f4eb2515909df76101b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42275 } } peers { permanent_uuid: "810a404a0f1d4d6991116bdddec2ea3d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40359 } }
I20260927 18:38:44.455138 7057 raft_consensus.cc:374] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:44.455235 7057 raft_consensus.cc:729] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810a404a0f1d4d6991116bdddec2ea3d, State: Initialized, Role: FOLLOWER
I20260927 18:38:44.455921 7057 consensus_queue.cc:260] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40271 } } peers { permanent_uuid: "1b190d56ff344f4eb2515909df76101b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42275 } } peers { permanent_uuid: "810a404a0f1d4d6991116bdddec2ea3d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40359 } }
I20260927 18:38:44.456979 7038 heartbeater.cc:503] Master 127.2.95.126:40399 was elected leader, sending a full tablet report...
I20260927 18:38:44.457751 7057 ts_tablet_manager.cc:1428] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260927 18:38:44.460919 7059 raft_consensus.cc:348] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40271 } } peers { permanent_uuid: "1b190d56ff344f4eb2515909df76101b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42275 } } peers { permanent_uuid: "810a404a0f1d4d6991116bdddec2ea3d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40359 } }
I20260927 18:38:44.461411 7059 raft_consensus.cc:374] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:44.461532 7059 raft_consensus.cc:729] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b190d56ff344f4eb2515909df76101b, State: Initialized, Role: FOLLOWER
I20260927 18:38:44.462174 7059 consensus_queue.cc:260] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40271 } } peers { permanent_uuid: "1b190d56ff344f4eb2515909df76101b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42275 } } peers { permanent_uuid: "810a404a0f1d4d6991116bdddec2ea3d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40359 } }
I20260927 18:38:44.463812 7059 ts_tablet_manager.cc:1428] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
W20260927 18:38:44.536651 6774 tablet.cc:2369] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:38:44.543524 7039 tablet.cc:2369] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:38:44.545192 6907 tablet.cc:2369] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:38:44.752002 7063 raft_consensus.cc:482] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:44.752377 7063 raft_consensus.cc:504] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40271 } } peers { permanent_uuid: "1b190d56ff344f4eb2515909df76101b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42275 } } peers { permanent_uuid: "810a404a0f1d4d6991116bdddec2ea3d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40359 } }
I20260927 18:38:44.754242 7063 leader_election.cc:288] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ff2ac78883c4886ba4f761ff2d8a6af (127.2.95.66:40271), 1b190d56ff344f4eb2515909df76101b (127.2.95.65:42275)
I20260927 18:38:44.762392 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.754869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:59221 (local address 127.2.95.65:42275)
0927 18:38:44.755053 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:44.755059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:44.755813 (+ 754us) server_negotiation.cc:407] Connection header received
0927 18:38:44.755940 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:44.755944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:44.756048 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:44.756172 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:44.756762 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.757634 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.760047 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.761279 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.761404 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.761504 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:44.761874 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 18:38:44.762077 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:38:44.762403 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.754522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:40271 (local address 127.2.95.67:46979)
0927 18:38:44.754896 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:44.754916 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:44.755060 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:44.755624 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:44.755630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:44.755662 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:44.755959 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:44.755970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.757349 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.757354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:44.759592 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:44.759610 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.761381 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.761388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:44.761611 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.761650 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 18:38:44.761801 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 18:38:44.761947 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":52}
I20260927 18:38:44.762627 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.754694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:46979 (local address 127.2.95.66:40271)
0927 18:38:44.754967 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:44.754974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:44.755004 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:38:44.755332 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:44.755341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:44.755445 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:44.755566 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:38:44.756109 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.757180 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.759796 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.761167 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.761329 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.761448 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:44.762129 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 18:38:44.762389 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 18:38:44.762647 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.754959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:42275 (local address 127.2.95.67:59221)
0927 18:38:44.755680 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:44.755698 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:44.755816 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:44.756244 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:44.756248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:44.756270 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:38:44.756591 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:44.756600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.757823 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.757829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:44.759875 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:44.759884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.761490 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.761497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:44.761644 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.761700 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 18:38:44.761801 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:38:44.761947 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":20,"thread_start_us":388,"threads_started":1}
I20260927 18:38:44.764075 6861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb20fcf28d14fdf9d6bb6160b9c039b" candidate_uuid: "810a404a0f1d4d6991116bdddec2ea3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af" is_pre_election: true
I20260927 18:38:44.764364 6728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb20fcf28d14fdf9d6bb6160b9c039b" candidate_uuid: "810a404a0f1d4d6991116bdddec2ea3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b190d56ff344f4eb2515909df76101b" is_pre_election: true
I20260927 18:38:44.764602 6861 raft_consensus.cc:2454] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 810a404a0f1d4d6991116bdddec2ea3d in term 0.
I20260927 18:38:44.764998 6728 raft_consensus.cc:2454] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 810a404a0f1d4d6991116bdddec2ea3d in term 0.
I20260927 18:38:44.765384 6929 leader_election.cc:302] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 810a404a0f1d4d6991116bdddec2ea3d, 8ff2ac78883c4886ba4f761ff2d8a6af; no voters:
I20260927 18:38:44.765892 7063 raft_consensus.cc:2781] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:44.766040 7063 raft_consensus.cc:482] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:44.766148 7063 raft_consensus.cc:3037] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:44.768580 7063 raft_consensus.cc:504] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40271 } } peers { permanent_uuid: "1b190d56ff344f4eb2515909df76101b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42275 } } peers { permanent_uuid: "810a404a0f1d4d6991116bdddec2ea3d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40359 } }
I20260927 18:38:44.769196 7063 leader_election.cc:288] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [CANDIDATE]: Term 1 election: Requested vote from peers 8ff2ac78883c4886ba4f761ff2d8a6af (127.2.95.66:40271), 1b190d56ff344f4eb2515909df76101b (127.2.95.65:42275)
I20260927 18:38:44.769716 6861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb20fcf28d14fdf9d6bb6160b9c039b" candidate_uuid: "810a404a0f1d4d6991116bdddec2ea3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af"
I20260927 18:38:44.769933 6861 raft_consensus.cc:3037] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:44.769989 6728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb20fcf28d14fdf9d6bb6160b9c039b" candidate_uuid: "810a404a0f1d4d6991116bdddec2ea3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b190d56ff344f4eb2515909df76101b"
I20260927 18:38:44.770260 6728 raft_consensus.cc:3037] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:44.772127 6861 raft_consensus.cc:2454] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 810a404a0f1d4d6991116bdddec2ea3d in term 1.
I20260927 18:38:44.772537 6929 leader_election.cc:302] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 810a404a0f1d4d6991116bdddec2ea3d, 8ff2ac78883c4886ba4f761ff2d8a6af; no voters:
I20260927 18:38:44.772791 7063 raft_consensus.cc:2781] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:44.773025 6728 raft_consensus.cc:2454] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 810a404a0f1d4d6991116bdddec2ea3d in term 1.
I20260927 18:38:44.773241 7063 raft_consensus.cc:686] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 1 LEADER]: Becoming Leader. State: Replica: 810a404a0f1d4d6991116bdddec2ea3d, State: Running, Role: LEADER
I20260927 18:38:44.773838 7063 consensus_queue.cc:237] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40271 } } peers { permanent_uuid: "1b190d56ff344f4eb2515909df76101b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42275 } } peers { permanent_uuid: "810a404a0f1d4d6991116bdddec2ea3d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40359 } }
I20260927 18:38:44.781039 6585 catalog_manager.cc:5495] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 810a404a0f1d4d6991116bdddec2ea3d (127.2.95.67). New cstate: current_term: 1 leader_uuid: "810a404a0f1d4d6991116bdddec2ea3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b190d56ff344f4eb2515909df76101b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "810a404a0f1d4d6991116bdddec2ea3d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 40359 } health_report { overall_health: HEALTHY } } }
I20260927 18:38:44.872251 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.865662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.2.95.67:40359)
0927 18:38:44.865975 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:44.865983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:44.866249 (+ 266us) server_negotiation.cc:407] Connection header received
0927 18:38:44.866505 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:44.866510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:44.866603 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:44.866766 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:38:44.867219 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.868211 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.869805 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.870236 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.870352 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.870937 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:38:44.871607 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:38:44.871765 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:44.871856 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 18:38:44.871956 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:38:44.882843 6861 raft_consensus.cc:1261] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af [term 1 FOLLOWER]: Refusing update from remote peer 810a404a0f1d4d6991116bdddec2ea3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:44.882843 6728 raft_consensus.cc:1261] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b [term 1 FOLLOWER]: Refusing update from remote peer 810a404a0f1d4d6991116bdddec2ea3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:44.884166 7066 consensus_queue.cc:1035] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ff2ac78883c4886ba4f761ff2d8a6af" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:44.884680 7063 consensus_queue.cc:1035] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b190d56ff344f4eb2515909df76101b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 42275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:44.903244 7070 mvcc.cc:205] Tried to move back new op lower bound from 7334028594707337216 to 7334028594277953536. Current Snapshot: MvccSnapshot[applied={T|T < 7334028594707337216}]
I20260927 18:38:44.927769 6584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:44.928184 6584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:44.933445 6584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf6e7bb4c56c4c358602cce9473f57b8: Sending DeleteTablet for 3 replicas of tablet bdb20fcf28d14fdf9d6bb6160b9c039b
I20260927 18:38:44.935012 6708 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb20fcf28d14fdf9d6bb6160b9c039b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:44 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:58930
I20260927 18:38:44.935266 6973 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb20fcf28d14fdf9d6bb6160b9c039b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:44 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:37534
I20260927 18:38:44.936177 6841 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb20fcf28d14fdf9d6bb6160b9c039b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:44 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:46096
I20260927 18:38:44.936383 7081 tablet_replica.cc:330] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b: stopping tablet replica
I20260927 18:38:44.937026 7081 raft_consensus.cc:2229] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:44.937165 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 6646
I20260927 18:38:44.937790 7081 raft_consensus.cc:2258] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:44.940601 7081 ts_tablet_manager.cc:1905] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:44.945030 7083 tablet_replica.cc:330] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af: stopping tablet replica
I20260927 18:38:44.945464 7082 tablet_replica.cc:330] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d: stopping tablet replica
I20260927 18:38:44.945751 7083 raft_consensus.cc:2229] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:44.946188 7082 raft_consensus.cc:2229] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:38:44.946378 7083 raft_consensus.cc:2258] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:44.947152 7082 raft_consensus.cc:2258] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:44.948284 7083 ts_tablet_manager.cc:1905] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:44.949604 7082 ts_tablet_manager.cc:1905] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:44.955583 7083 ts_tablet_manager.cc:1918] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:44.955863 7083 log.cc:1196] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/bdb20fcf28d14fdf9d6bb6160b9c039b
I20260927 18:38:44.956449 7083 ts_tablet_manager.cc:1939] T bdb20fcf28d14fdf9d6bb6160b9c039b P 8ff2ac78883c4886ba4f761ff2d8a6af: Deleting consensus metadata
I20260927 18:38:44.956429 7081 ts_tablet_manager.cc:1918] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:44.956593 7081 log.cc:1196] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/bdb20fcf28d14fdf9d6bb6160b9c039b
I20260927 18:38:44.958540 7082 ts_tablet_manager.cc:1918] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:44.958642 6572 catalog_manager.cc:4841] TS 8ff2ac78883c4886ba4f761ff2d8a6af (127.2.95.66:40271): tablet bdb20fcf28d14fdf9d6bb6160b9c039b (table test-table [id=f45b0b3f383d4d5a8f3c79fe62b692df]) successfully deleted
I20260927 18:38:44.958866 7082 log.cc:1196] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/bdb20fcf28d14fdf9d6bb6160b9c039b
I20260927 18:38:44.959550 7082 ts_tablet_manager.cc:1939] T bdb20fcf28d14fdf9d6bb6160b9c039b P 810a404a0f1d4d6991116bdddec2ea3d: Deleting consensus metadata
Sep 27 18:38:44 dist-test-slave-9jll krb5kdc[6541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:44.961098 7081 ts_tablet_manager.cc:1939] T bdb20fcf28d14fdf9d6bb6160b9c039b P 1b190d56ff344f4eb2515909df76101b: Deleting consensus metadata
I20260927 18:38:44.961846 6569 catalog_manager.cc:4841] TS 810a404a0f1d4d6991116bdddec2ea3d (127.2.95.67:40359): tablet bdb20fcf28d14fdf9d6bb6160b9c039b (table test-table [id=f45b0b3f383d4d5a8f3c79fe62b692df]) successfully deleted
I20260927 18:38:44.963053 6571 catalog_manager.cc:4841] TS 1b190d56ff344f4eb2515909df76101b (127.2.95.65:42275): tablet bdb20fcf28d14fdf9d6bb6160b9c039b (table test-table [id=f45b0b3f383d4d5a8f3c79fe62b692df]) successfully deleted
I20260927 18:38:44.971854 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:44.938821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58938 (local address 127.2.95.65:42275)
0927 18:38:44.939095 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:44.939101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:44.942428 (+ 3327us) server_negotiation.cc:407] Connection header received
0927 18:38:44.944243 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:44.944249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:44.944367 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:44.944451 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:44.947904 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.948954 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.952638 (+ 3684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:44.953292 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:44.953394 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:44.964073 (+ 10679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:44.964094 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:44.964119 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:44.964185 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:44.968444 (+ 4259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:44.969383 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:44.969394 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:44.969398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:44.969592 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:44.970028 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:44.970034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:44.970038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:44.970600 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:44.970968 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:44.971150 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 18:38:44.971410 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260927 18:38:44.972678 6768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:45.064010 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 6646
I20260927 18:38:45.078576 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 6779
Sep 27 18:38:45 dist-test-slave-9jll krb5kdc[6541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:38:45.096400 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:45.079601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.2.95.66:40271)
0927 18:38:45.079781 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:45.079785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:45.079856 (+ 71us) server_negotiation.cc:407] Connection header received
0927 18:38:45.080085 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:45.080090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:45.080162 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:45.080255 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:45.082048 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:45.082823 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:45.084575 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:45.085014 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:45.085117 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:45.090712 (+ 5595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:45.090729 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:45.090750 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:45.090791 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:45.093491 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:45.094342 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:45.094349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:45.094352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:45.094485 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:45.094903 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:45.094906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:45.094909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:45.095274 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:45.095525 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:45.095932 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 18:38:45.096107 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:38:45.097318 6901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:45.186801 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 6779
I20260927 18:38:45.200911 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 6911
Sep 27 18:38:45 dist-test-slave-9jll krb5kdc[6541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:38:45.220300 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:45.201809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.2.95.67:40359)
0927 18:38:45.202035 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:45.202039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:45.202309 (+ 270us) server_negotiation.cc:407] Connection header received
0927 18:38:45.202527 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:45.202531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:45.202615 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:45.202735 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:45.204655 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:45.205573 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:45.207461 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:45.207931 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:45.208102 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:45.214010 (+ 5908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:45.214027 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:45.214045 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:45.214091 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:45.217143 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:45.218114 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:45.218120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:45.218123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:45.218305 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:45.218736 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:45.218739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:45.218741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:45.219126 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:45.219354 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:45.219759 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 18:38:45.219953 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:38:45.221221 7033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:45.341425 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 6911
I20260927 18:38:45.359761 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 6554
I20260927 18:38:45.361131 6615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:45.486260 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 6554
2026-09-27T18:38:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4466 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T18:38:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:45Z Disabled control of system clock
I20260927 18:38:45.555684 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33779
--webserver_interface=127.2.95.126
--webserver_port=0
--builtin_ntp_servers=127.2.95.84:36805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:33779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:38:45.894819 7092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:45.895149 7092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:45.895277 7092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:45.902549 7092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:45.902669 7092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:45.902707 7092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:45.902761 7092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:45.902803 7092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:45.912206 7092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:33779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:45.913565 7092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:45.915344 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:45.927413 7098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:45.927449 7097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:45.927475 7100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:45.928781 7092 server_base.cc:1034] running on GCE node
I20260927 18:38:45.929646 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:45.931428 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:45.932689 7092 hybrid_clock.cc:648] HybridClock initialized: now 1790534325932641 us; error 43 us; skew 500 ppm
I20260927 18:38:45.933135 7092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:38:45.935636 7092 webserver.cc:460] Webserver started at http://127.2.95.126:40653/ using document root <none> and password file <none>
I20260927 18:38:45.936389 7092 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:45.936493 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:45.936815 7092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:45.940074 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "ff32bc7107264dc183c633d5501b3786"
format_stamp: "Formatted at 2026-09-27 18:38:45 on dist-test-slave-9jll"
server_key: "34beaa04d329672fefec87d313ec9a6b"
server_key_iv: "0e34ef626db6b488dd3cd10742b0b7da"
server_key_version: "encryptionkey@0"
I20260927 18:38:45.940914 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "ff32bc7107264dc183c633d5501b3786"
format_stamp: "Formatted at 2026-09-27 18:38:45 on dist-test-slave-9jll"
server_key: "34beaa04d329672fefec87d313ec9a6b"
server_key_iv: "0e34ef626db6b488dd3cd10742b0b7da"
server_key_version: "encryptionkey@0"
I20260927 18:38:45.946764 7092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:38:45.950695 7106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:45.952473 7092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 18:38:45.952663 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "ff32bc7107264dc183c633d5501b3786"
format_stamp: "Formatted at 2026-09-27 18:38:45 on dist-test-slave-9jll"
server_key: "34beaa04d329672fefec87d313ec9a6b"
server_key_iv: "0e34ef626db6b488dd3cd10742b0b7da"
server_key_version: "encryptionkey@0"
I20260927 18:38:45.952889 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:45.959131 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:45.959991 7092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:45.960279 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:45.978080 7092 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:33779
I20260927 18:38:45.978138 7157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:33779 every 8 connection(s)
I20260927 18:38:45.979427 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:45.983346 7158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:45.987318 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 7092
I20260927 18:38:45.987521 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:45.988312 2429 external_mini_cluster.cc:1428] Setting key 1e94802ef9034d05c5c6adf939c6b041
I20260927 18:38:45.995520 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786: Bootstrap starting.
I20260927 18:38:45.997921 7161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:45.990670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.2.95.126:33779)
0927 18:38:45.991664 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:45.991680 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:45.991727 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:38:45.993853 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:45.993883 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:45.994352 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:45.994709 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:45.996001 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:45.996009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:45.996023 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:45.996044 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:45.996115 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:45.996131 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:45.996340 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:45.996450 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 18:38:45.996700 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":340,"threads_started":1}
I20260927 18:38:45.998967 7158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:46.000254 7158 log.cc:824] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:46.003844 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786: No bootstrap required, opened a new log
I20260927 18:38:46.011688 7158 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff32bc7107264dc183c633d5501b3786" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33779 } }
I20260927 18:38:46.012099 7158 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:46.012172 7158 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff32bc7107264dc183c633d5501b3786, State: Initialized, Role: FOLLOWER
I20260927 18:38:46.012707 7158 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff32bc7107264dc183c633d5501b3786" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33779 } }
I20260927 18:38:46.012929 7158 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:46.013031 7158 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:46.013159 7158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:46.015378 7158 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff32bc7107264dc183c633d5501b3786" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33779 } }
I20260927 18:38:46.015900 7158 leader_election.cc:302] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff32bc7107264dc183c633d5501b3786; no voters:
I20260927 18:38:46.016566 7158 leader_election.cc:288] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:46.016705 7163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:46.017988 7163 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [term 1 LEADER]: Becoming Leader. State: Replica: ff32bc7107264dc183c633d5501b3786, State: Running, Role: LEADER
I20260927 18:38:46.018573 7163 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff32bc7107264dc183c633d5501b3786" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33779 } }
I20260927 18:38:46.018806 7158 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:46.021340 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff32bc7107264dc183c633d5501b3786. Latest consensus state: current_term: 1 leader_uuid: "ff32bc7107264dc183c633d5501b3786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff32bc7107264dc183c633d5501b3786" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33779 } } }
I20260927 18:38:46.021319 7164 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff32bc7107264dc183c633d5501b3786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff32bc7107264dc183c633d5501b3786" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33779 } } }
I20260927 18:38:46.021808 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:46.021806 7164 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:46.023972 7170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:46.029374 7170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:46.039243 7170 catalog_manager.cc:1347] Generated new cluster ID: b7d2c1d309d141ca8c3a4772a2b39b93
I20260927 18:38:46.039373 7170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:46.075779 7170 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:46.076247 7170 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:46.087303 7170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786: Generated new TSK 0
I20260927 18:38:46.088196 7170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:46.098613 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:33779
--builtin_ntp_servers=127.2.95.84:36805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 18:38:46.476739 7182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:46.477036 7182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:46.477149 7182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:46.484598 7182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:46.484717 7182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:46.484860 7182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:38:46.494180 7182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:33779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:46.495486 7182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:46.497365 7182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:46.510957 7188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:46.511031 7190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:46.510972 7187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:46.511767 7182 server_base.cc:1034] running on GCE node
I20260927 18:38:46.512421 7182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:46.513486 7182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:46.514706 7182 hybrid_clock.cc:648] HybridClock initialized: now 1790534326514643 us; error 63 us; skew 500 ppm
I20260927 18:38:46.515131 7182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:38:46.517616 7182 webserver.cc:460] Webserver started at http://127.2.95.65:38597/ using document root <none> and password file <none>
I20260927 18:38:46.518420 7182 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:46.518532 7182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:46.518893 7182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:46.522113 7182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "92a6fdd29cef4d9a8e43abe157f990b4"
format_stamp: "Formatted at 2026-09-27 18:38:46 on dist-test-slave-9jll"
server_key: "7c975415bd12a5fe0ff0976200f3187b"
server_key_iv: "1b1ead74bf949faf25ad9e13ed502b77"
server_key_version: "encryptionkey@0"
I20260927 18:38:46.523116 7182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "92a6fdd29cef4d9a8e43abe157f990b4"
format_stamp: "Formatted at 2026-09-27 18:38:46 on dist-test-slave-9jll"
server_key: "7c975415bd12a5fe0ff0976200f3187b"
server_key_iv: "1b1ead74bf949faf25ad9e13ed502b77"
server_key_version: "encryptionkey@0"
I20260927 18:38:46.529067 7182 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.006s
I20260927 18:38:46.533265 7196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:46.535126 7182 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 18:38:46.535327 7182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "92a6fdd29cef4d9a8e43abe157f990b4"
format_stamp: "Formatted at 2026-09-27 18:38:46 on dist-test-slave-9jll"
server_key: "7c975415bd12a5fe0ff0976200f3187b"
server_key_iv: "1b1ead74bf949faf25ad9e13ed502b77"
server_key_version: "encryptionkey@0"
I20260927 18:38:46.535578 7182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:46.541813 7182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:46.542743 7182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:46.543022 7182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:46.544142 7182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:46.545933 7182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:46.546038 7182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:46.546147 7182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:46.546213 7182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:46.578336 7182 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:34675
I20260927 18:38:46.578435 7308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:34675 every 8 connection(s)
I20260927 18:38:46.579696 7182 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:46.582397 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 7182
I20260927 18:38:46.582623 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:46.583370 2429 external_mini_cluster.cc:1428] Setting key 56bd7e3f97388fd425dabd482ad93251
I20260927 18:38:46.586047 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:33779
--builtin_ntp_servers=127.2.95.84:36805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 18:38:46.588773 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:46.582583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:60221 (local address 127.2.95.126:33779)
0927 18:38:46.583121 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:46.583129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:46.584413 (+ 1284us) server_negotiation.cc:407] Connection header received
0927 18:38:46.586165 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:46.586170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:46.586238 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:46.586352 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:46.588060 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:46.588065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:46.588068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:46.588089 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:46.588132 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:46.588145 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:46.588255 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:46.588484 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 18:38:46.588543 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":252,"threads_started":1}
I20260927 18:38:46.590144 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:46.583185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:33779 (local address 127.2.95.65:60221)
0927 18:38:46.584253 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:46.584311 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:46.585891 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:46.586673 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:46.586688 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:46.586858 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:46.587655 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:46.587659 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:46.587715 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:46.587837 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:46.588240 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:46.588249 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:46.588257 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:38:46.588532 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 18:38:46.588813 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":309,"threads_started":1}
I20260927 18:38:46.591836 7309 heartbeater.cc:347] Connected to a master server at 127.2.95.126:33779
I20260927 18:38:46.592135 7309 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:46.592726 7309 heartbeater.cc:511] Master 127.2.95.126:33779 requested a full tablet report, sending...
I20260927 18:38:46.594974 7123 ts_manager.cc:194] Registered new tserver with Master: 92a6fdd29cef4d9a8e43abe157f990b4 (127.2.95.65:34675)
W20260927 18:38:46.915136 7313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:46.915483 7313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:46.915619 7313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:46.923786 7313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:46.924000 7313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:46.924201 7313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:38:46.933853 7313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:33779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:46.935108 7313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:46.936885 7313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:46.949918 7321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:46.950081 7318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:46.950105 7319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:46.951258 7313 server_base.cc:1034] running on GCE node
I20260927 18:38:46.951951 7313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:46.952973 7313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:46.954192 7313 hybrid_clock.cc:648] HybridClock initialized: now 1790534326954137 us; error 46 us; skew 500 ppm
I20260927 18:38:46.954542 7313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:38:46.957058 7313 webserver.cc:460] Webserver started at http://127.2.95.66:36267/ using document root <none> and password file <none>
I20260927 18:38:46.957890 7313 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:46.958009 7313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:46.958328 7313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:46.961354 7313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "739c7a2eab454f1a8e6fe3487b48f488"
format_stamp: "Formatted at 2026-09-27 18:38:46 on dist-test-slave-9jll"
server_key: "0673b0bc2485dc2dcbea0bc5895aca6f"
server_key_iv: "c5b55f35513e7d349ed2186ec8f367fa"
server_key_version: "encryptionkey@0"
I20260927 18:38:46.962160 7313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "739c7a2eab454f1a8e6fe3487b48f488"
format_stamp: "Formatted at 2026-09-27 18:38:46 on dist-test-slave-9jll"
server_key: "0673b0bc2485dc2dcbea0bc5895aca6f"
server_key_iv: "c5b55f35513e7d349ed2186ec8f367fa"
server_key_version: "encryptionkey@0"
I20260927 18:38:46.967553 7313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 18:38:46.971419 7327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:46.972883 7313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:38:46.973119 7313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "739c7a2eab454f1a8e6fe3487b48f488"
format_stamp: "Formatted at 2026-09-27 18:38:46 on dist-test-slave-9jll"
server_key: "0673b0bc2485dc2dcbea0bc5895aca6f"
server_key_iv: "c5b55f35513e7d349ed2186ec8f367fa"
server_key_version: "encryptionkey@0"
I20260927 18:38:46.973357 7313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:46.980134 7313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:46.981261 7313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:46.981576 7313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:46.982784 7313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:46.984676 7313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:46.984788 7313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:46.984892 7313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:46.984961 7313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:47.017284 7313 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:41743
I20260927 18:38:47.017330 7439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:41743 every 8 connection(s)
I20260927 18:38:47.018651 7313 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:47.027431 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 7313
I20260927 18:38:47.027601 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:47.027613 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.021474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:33707 (local address 127.2.95.126:33779)
0927 18:38:47.021754 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:47.021758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:47.023333 (+ 1575us) server_negotiation.cc:407] Connection header received
0927 18:38:47.025142 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:47.025146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:47.025220 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:47.025314 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:47.026788 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:47.026793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:47.026796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:47.026814 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:47.026845 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:47.026857 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:47.026968 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:47.027308 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 18:38:47.027373 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 18:38:47.028138 2429 external_mini_cluster.cc:1428] Setting key 2c599a960eaff607e1c021efa370e045
I20260927 18:38:47.028885 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.022115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:33779 (local address 127.2.95.66:33707)
0927 18:38:47.023157 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:47.023196 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:47.024878 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:47.025624 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:47.025633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:47.025797 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:47.026485 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:47.026488 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:47.026526 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:47.026613 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:47.026978 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:47.026990 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:47.026999 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:38:47.027312 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 18:38:47.027652 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":287,"threads_started":1}
I20260927 18:38:47.030334 7440 heartbeater.cc:347] Connected to a master server at 127.2.95.126:33779
I20260927 18:38:47.030682 7440 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:47.031293 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:33779
--builtin_ntp_servers=127.2.95.84:36805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 18:38:47.031435 7440 heartbeater.cc:511] Master 127.2.95.126:33779 requested a full tablet report, sending...
I20260927 18:38:47.033008 7123 ts_manager.cc:194] Registered new tserver with Master: 739c7a2eab454f1a8e6fe3487b48f488 (127.2.95.66:41743)
W20260927 18:38:47.355046 7443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:47.355348 7443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:47.355456 7443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:47.363173 7443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:47.363291 7443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:47.363437 7443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:38:47.373440 7443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:33779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:47.374979 7443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:47.376871 7443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:47.390223 7448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:47.390170 7449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:47.390170 7451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:47.391227 7443 server_base.cc:1034] running on GCE node
I20260927 18:38:47.391898 7443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:47.392879 7443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:47.394121 7443 hybrid_clock.cc:648] HybridClock initialized: now 1790534327394074 us; error 33 us; skew 500 ppm
I20260927 18:38:47.394433 7443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:38:47.396795 7443 webserver.cc:460] Webserver started at http://127.2.95.67:35641/ using document root <none> and password file <none>
I20260927 18:38:47.397485 7443 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:47.397595 7443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:47.397931 7443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:47.400880 7443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "07ae9c675628499c8fffb0d2c6646d7e"
format_stamp: "Formatted at 2026-09-27 18:38:47 on dist-test-slave-9jll"
server_key: "479736a7e1f7e5a34f2e6b7395545776"
server_key_iv: "7e7a135f55ad52d165457664472cb768"
server_key_version: "encryptionkey@0"
I20260927 18:38:47.401659 7443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "07ae9c675628499c8fffb0d2c6646d7e"
format_stamp: "Formatted at 2026-09-27 18:38:47 on dist-test-slave-9jll"
server_key: "479736a7e1f7e5a34f2e6b7395545776"
server_key_iv: "7e7a135f55ad52d165457664472cb768"
server_key_version: "encryptionkey@0"
I20260927 18:38:47.407197 7443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 18:38:47.410914 7457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:47.412439 7443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:38:47.412616 7443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "07ae9c675628499c8fffb0d2c6646d7e"
format_stamp: "Formatted at 2026-09-27 18:38:47 on dist-test-slave-9jll"
server_key: "479736a7e1f7e5a34f2e6b7395545776"
server_key_iv: "7e7a135f55ad52d165457664472cb768"
server_key_version: "encryptionkey@0"
I20260927 18:38:47.412864 7443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:47.419044 7443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:47.419946 7443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:47.420204 7443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:47.421173 7443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:47.422699 7443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:47.422787 7443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:47.422900 7443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:47.422972 7443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:47.454617 7443 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:45863
I20260927 18:38:47.454689 7569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:45863 every 8 connection(s)
I20260927 18:38:47.455922 7443 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:38:47.461778 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 7443
I20260927 18:38:47.461953 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:47.462483 2429 external_mini_cluster.cc:1428] Setting key 6dbd1c8dcbddcf8965044159bf7e7d5c
I20260927 18:38:47.465212 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.458714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:59569 (local address 127.2.95.126:33779)
0927 18:38:47.458941 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:47.458945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:47.460486 (+ 1541us) server_negotiation.cc:407] Connection header received
0927 18:38:47.462555 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:47.462562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:47.462636 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:47.462731 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:47.464174 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:47.464179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:47.464182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:47.464213 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:47.464257 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:47.464273 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:47.464408 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:47.464694 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 18:38:47.464759 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 18:38:47.466301 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.459269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:33779 (local address 127.2.95.67:59569)
0927 18:38:47.460304 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:47.460342 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:47.462200 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:47.463032 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:47.463043 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:47.463150 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:47.463786 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:47.463789 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:47.463843 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:47.463968 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:47.464411 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:47.464424 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:47.464433 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:38:47.464733 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 18:38:47.465102 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":325,"threads_started":1}
I20260927 18:38:47.467852 7570 heartbeater.cc:347] Connected to a master server at 127.2.95.126:33779
I20260927 18:38:47.468160 7570 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:47.468739 7570 heartbeater.cc:511] Master 127.2.95.126:33779 requested a full tablet report, sending...
I20260927 18:38:47.470218 7123 ts_manager.cc:194] Registered new tserver with Master: 07ae9c675628499c8fffb0d2c6646d7e (127.2.95.67:45863)
I20260927 18:38:47.478827 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:38:47.486016 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.483632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34532 (local address 127.2.95.126:33779)
0927 18:38:47.483840 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:47.483848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:47.484257 (+ 409us) server_negotiation.cc:407] Connection header received
0927 18:38:47.484420 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:47.484424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:47.484480 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:47.484562 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:47.485431 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:47.485436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:47.485439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:47.485455 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:47.485489 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:47.485500 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:47.485608 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:47.485646 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 18:38:47.485734 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:38:47.490136 7123 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:38:47.497061 7123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:38:47.514134 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.509218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34675 (local address 127.0.0.1:59524)
0927 18:38:47.510040 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:47.510069 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:47.510216 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:47.511898 (+ 1682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:47.511908 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:47.512019 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:47.512815 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:47.512818 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:47.512846 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:47.512923 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:47.513434 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:47.513445 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:47.513452 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:38:47.513632 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 18:38:47.513747 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":410,"threads_started":1}
I20260927 18:38:47.514875 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.509447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.2.95.65:34675)
0927 18:38:47.509962 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:47.509970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:47.510190 (+ 220us) server_negotiation.cc:407] Connection header received
0927 18:38:47.510384 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:47.510392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:47.510632 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:47.510790 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:47.513127 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:47.513136 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:47.513151 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:47.513178 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:47.513238 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:47.513259 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:47.513385 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:47.514567 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0927 18:38:47.514653 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":768,"threads_started":1}
I20260927 18:38:47.517319 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.511051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:41743 (local address 127.0.0.1:59786)
0927 18:38:47.514330 (+ 3279us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:47.514347 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:47.514420 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:47.515539 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:47.515544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:47.515587 (+ 43us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:47.516179 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:47.516181 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:47.516198 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:47.516245 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:47.516916 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:47.516922 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:47.516927 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:38:47.516998 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 18:38:47.517060 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3191,"threads_started":1}
I20260927 18:38:47.518188 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.511622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.2.95.66:41743)
0927 18:38:47.513841 (+ 2219us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:47.513849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:47.515126 (+ 1277us) server_negotiation.cc:407] Connection header received
0927 18:38:47.515193 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:47.515200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:47.515384 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:47.515518 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:47.516683 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:47.516692 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:47.516706 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:47.516731 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:47.516787 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:47.516808 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:47.517768 (+ 960us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:47.517832 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 18:38:47.517912 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":1055,"threads_started":1}
I20260927 18:38:47.521562 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.510541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:45863 (local address 127.0.0.1:44668)
0927 18:38:47.511044 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:47.511061 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:47.511152 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:47.519646 (+ 8494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:47.519651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:47.519677 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:47.520293 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:47.520296 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:47.520314 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:47.520360 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:47.520974 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:47.520979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:47.520985 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:38:47.521110 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 18:38:47.521164 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":232,"threads_started":1}
I20260927 18:38:47.521569 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.510887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.2.95.67:45863)
0927 18:38:47.519127 (+ 8240us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:47.519135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:47.519171 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:38:47.519243 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:47.519251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:47.519478 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:47.519634 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:47.520725 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:47.520733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:47.520755 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:47.520773 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:47.520836 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:47.520856 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:47.521045 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:47.521111 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 18:38:47.521207 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8055,"thread_start_us":7915,"threads_started":1}
I20260927 18:38:47.522935 7375 tablet_service.cc:1467] Processing CreateTablet for tablet 7cdddddfb417443db167f923b076e551 (DEFAULT_TABLE table=test-table [id=189456d8bcba437fa84d6f165b26b740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:47.522945 7244 tablet_service.cc:1467] Processing CreateTablet for tablet 7cdddddfb417443db167f923b076e551 (DEFAULT_TABLE table=test-table [id=189456d8bcba437fa84d6f165b26b740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:47.524333 7375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cdddddfb417443db167f923b076e551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:47.524399 7244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cdddddfb417443db167f923b076e551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:47.525481 7505 tablet_service.cc:1467] Processing CreateTablet for tablet 7cdddddfb417443db167f923b076e551 (DEFAULT_TABLE table=test-table [id=189456d8bcba437fa84d6f165b26b740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:38:47.527025 7505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cdddddfb417443db167f923b076e551. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:47.536995 7588 tablet_bootstrap.cc:492] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488: Bootstrap starting.
I20260927 18:38:47.540400 7588 tablet_bootstrap.cc:654] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:47.541575 7588 log.cc:824] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:47.545065 7589 tablet_bootstrap.cc:492] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4: Bootstrap starting.
I20260927 18:38:47.548110 7590 tablet_bootstrap.cc:492] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e: Bootstrap starting.
I20260927 18:38:47.549268 7589 tablet_bootstrap.cc:654] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:47.550434 7588 tablet_bootstrap.cc:492] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488: No bootstrap required, opened a new log
I20260927 18:38:47.550949 7588 ts_tablet_manager.cc:1397] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.004s
I20260927 18:38:47.551267 7589 log.cc:824] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:47.551744 7590 tablet_bootstrap.cc:654] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:47.552793 7590 log.cc:824] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:47.554418 7589 tablet_bootstrap.cc:492] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4: No bootstrap required, opened a new log
I20260927 18:38:47.554685 7589 ts_tablet_manager.cc:1397] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260927 18:38:47.556300 7590 tablet_bootstrap.cc:492] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e: No bootstrap required, opened a new log
I20260927 18:38:47.556610 7590 ts_tablet_manager.cc:1397] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 18:38:47.559298 7588 raft_consensus.cc:348] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a6fdd29cef4d9a8e43abe157f990b4" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34675 } } peers { permanent_uuid: "07ae9c675628499c8fffb0d2c6646d7e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45863 } } peers { permanent_uuid: "739c7a2eab454f1a8e6fe3487b48f488" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41743 } }
I20260927 18:38:47.559684 7588 raft_consensus.cc:374] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:47.559897 7588 raft_consensus.cc:729] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739c7a2eab454f1a8e6fe3487b48f488, State: Initialized, Role: FOLLOWER
I20260927 18:38:47.560554 7588 consensus_queue.cc:260] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a6fdd29cef4d9a8e43abe157f990b4" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34675 } } peers { permanent_uuid: "07ae9c675628499c8fffb0d2c6646d7e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45863 } } peers { permanent_uuid: "739c7a2eab454f1a8e6fe3487b48f488" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41743 } }
I20260927 18:38:47.565135 7589 raft_consensus.cc:348] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a6fdd29cef4d9a8e43abe157f990b4" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34675 } } peers { permanent_uuid: "07ae9c675628499c8fffb0d2c6646d7e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45863 } } peers { permanent_uuid: "739c7a2eab454f1a8e6fe3487b48f488" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41743 } }
I20260927 18:38:47.565636 7589 raft_consensus.cc:374] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:47.565775 7589 raft_consensus.cc:729] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a6fdd29cef4d9a8e43abe157f990b4, State: Initialized, Role: FOLLOWER
I20260927 18:38:47.566604 7440 heartbeater.cc:503] Master 127.2.95.126:33779 was elected leader, sending a full tablet report...
I20260927 18:38:47.566571 7589 consensus_queue.cc:260] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a6fdd29cef4d9a8e43abe157f990b4" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34675 } } peers { permanent_uuid: "07ae9c675628499c8fffb0d2c6646d7e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45863 } } peers { permanent_uuid: "739c7a2eab454f1a8e6fe3487b48f488" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41743 } }
I20260927 18:38:47.566735 7590 raft_consensus.cc:348] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a6fdd29cef4d9a8e43abe157f990b4" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34675 } } peers { permanent_uuid: "07ae9c675628499c8fffb0d2c6646d7e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45863 } } peers { permanent_uuid: "739c7a2eab454f1a8e6fe3487b48f488" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41743 } }
I20260927 18:38:47.567149 7590 raft_consensus.cc:374] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:47.567250 7590 raft_consensus.cc:729] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae9c675628499c8fffb0d2c6646d7e, State: Initialized, Role: FOLLOWER
I20260927 18:38:47.567291 7588 ts_tablet_manager.cc:1428] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260927 18:38:47.568409 7309 heartbeater.cc:503] Master 127.2.95.126:33779 was elected leader, sending a full tablet report...
I20260927 18:38:47.569096 7589 ts_tablet_manager.cc:1428] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260927 18:38:47.568006 7590 consensus_queue.cc:260] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a6fdd29cef4d9a8e43abe157f990b4" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34675 } } peers { permanent_uuid: "07ae9c675628499c8fffb0d2c6646d7e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45863 } } peers { permanent_uuid: "739c7a2eab454f1a8e6fe3487b48f488" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41743 } }
I20260927 18:38:47.570300 7570 heartbeater.cc:503] Master 127.2.95.126:33779 was elected leader, sending a full tablet report...
I20260927 18:38:47.572399 7590 ts_tablet_manager.cc:1428] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
W20260927 18:38:47.582033 7310 tablet.cc:2369] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:38:47.651247 7594 raft_consensus.cc:482] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:38:47.651561 7594 raft_consensus.cc:504] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a6fdd29cef4d9a8e43abe157f990b4" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34675 } } peers { permanent_uuid: "07ae9c675628499c8fffb0d2c6646d7e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45863 } } peers { permanent_uuid: "739c7a2eab454f1a8e6fe3487b48f488" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41743 } }
I20260927 18:38:47.653213 7594 leader_election.cc:288] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92a6fdd29cef4d9a8e43abe157f990b4 (127.2.95.65:34675), 07ae9c675628499c8fffb0d2c6646d7e (127.2.95.67:45863)
I20260927 18:38:47.656472 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.653777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:43721 (local address 127.2.95.65:34675)
0927 18:38:47.653929 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:47.653934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:47.654319 (+ 385us) server_negotiation.cc:407] Connection header received
0927 18:38:47.654523 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:47.654529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:47.654594 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:47.654705 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:47.655855 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:47.655859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:47.655862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:47.655877 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:47.655906 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:47.655917 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:47.656092 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:47.656144 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 18:38:47.656201 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 18:38:47.656718 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.653558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34675 (local address 127.2.95.66:43721)
0927 18:38:47.654231 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:47.654249 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:47.654368 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:47.654869 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:47.654872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:47.654893 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:47.655660 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:47.655663 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:47.655676 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:47.655716 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:47.655996 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:47.656000 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:47.656005 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:38:47.656143 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 18:38:47.656217 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":345,"threads_started":1}
I20260927 18:38:47.656945 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.654597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:56335 (local address 127.2.95.67:45863)
0927 18:38:47.654768 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:47.654774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:47.655052 (+ 278us) server_negotiation.cc:407] Connection header received
0927 18:38:47.655153 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:47.655159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:47.655217 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:47.655305 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:47.656364 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:47.656370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:47.656375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:47.656393 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:47.656422 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:47.656433 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:47.656537 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:47.656650 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 18:38:47.656710 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 18:38:47.657590 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.654391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:45863 (local address 127.2.95.66:56335)
0927 18:38:47.654937 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:47.654958 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:47.655058 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:47.655403 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:47.655408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:47.655432 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:47.656123 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:38:47.656126 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:47.656138 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:38:47.656172 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:47.656520 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:47.656525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:47.656530 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 18:38:47.657213 (+ 683us) client_negotiation.cc:240] Negotiation successful
0927 18:38:47.657274 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":265,"threads_started":1}
I20260927 18:38:47.657766 7264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cdddddfb417443db167f923b076e551" candidate_uuid: "739c7a2eab454f1a8e6fe3487b48f488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a6fdd29cef4d9a8e43abe157f990b4" is_pre_election: true
I20260927 18:38:47.658344 7264 raft_consensus.cc:2454] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 739c7a2eab454f1a8e6fe3487b48f488 in term 0.
I20260927 18:38:47.658496 7525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cdddddfb417443db167f923b076e551" candidate_uuid: "739c7a2eab454f1a8e6fe3487b48f488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae9c675628499c8fffb0d2c6646d7e" is_pre_election: true
I20260927 18:38:47.658989 7525 raft_consensus.cc:2454] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 739c7a2eab454f1a8e6fe3487b48f488 in term 0.
I20260927 18:38:47.659011 7329 leader_election.cc:302] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 739c7a2eab454f1a8e6fe3487b48f488, 92a6fdd29cef4d9a8e43abe157f990b4; no voters:
I20260927 18:38:47.659368 7594 raft_consensus.cc:2781] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:38:47.659525 7594 raft_consensus.cc:482] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:38:47.659637 7594 raft_consensus.cc:3037] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:47.661837 7594 raft_consensus.cc:504] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a6fdd29cef4d9a8e43abe157f990b4" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34675 } } peers { permanent_uuid: "07ae9c675628499c8fffb0d2c6646d7e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45863 } } peers { permanent_uuid: "739c7a2eab454f1a8e6fe3487b48f488" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41743 } }
I20260927 18:38:47.662405 7594 leader_election.cc:288] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [CANDIDATE]: Term 1 election: Requested vote from peers 92a6fdd29cef4d9a8e43abe157f990b4 (127.2.95.65:34675), 07ae9c675628499c8fffb0d2c6646d7e (127.2.95.67:45863)
I20260927 18:38:47.662925 7264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cdddddfb417443db167f923b076e551" candidate_uuid: "739c7a2eab454f1a8e6fe3487b48f488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a6fdd29cef4d9a8e43abe157f990b4"
I20260927 18:38:47.662948 7525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cdddddfb417443db167f923b076e551" candidate_uuid: "739c7a2eab454f1a8e6fe3487b48f488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae9c675628499c8fffb0d2c6646d7e"
I20260927 18:38:47.663115 7264 raft_consensus.cc:3037] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:47.663126 7525 raft_consensus.cc:3037] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:47.665441 7525 raft_consensus.cc:2454] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 739c7a2eab454f1a8e6fe3487b48f488 in term 1.
I20260927 18:38:47.665436 7264 raft_consensus.cc:2454] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 739c7a2eab454f1a8e6fe3487b48f488 in term 1.
I20260927 18:38:47.665908 7329 leader_election.cc:302] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae9c675628499c8fffb0d2c6646d7e, 739c7a2eab454f1a8e6fe3487b48f488; no voters:
I20260927 18:38:47.666216 7594 raft_consensus.cc:2781] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:47.666663 7594 raft_consensus.cc:686] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 1 LEADER]: Becoming Leader. State: Replica: 739c7a2eab454f1a8e6fe3487b48f488, State: Running, Role: LEADER
I20260927 18:38:47.667156 7594 consensus_queue.cc:237] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a6fdd29cef4d9a8e43abe157f990b4" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34675 } } peers { permanent_uuid: "07ae9c675628499c8fffb0d2c6646d7e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45863 } } peers { permanent_uuid: "739c7a2eab454f1a8e6fe3487b48f488" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41743 } }
I20260927 18:38:47.672297 7121 catalog_manager.cc:5495] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 739c7a2eab454f1a8e6fe3487b48f488 (127.2.95.66). New cstate: current_term: 1 leader_uuid: "739c7a2eab454f1a8e6fe3487b48f488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a6fdd29cef4d9a8e43abe157f990b4" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae9c675628499c8fffb0d2c6646d7e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "739c7a2eab454f1a8e6fe3487b48f488" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41743 } health_report { overall_health: HEALTHY } } }
W20260927 18:38:47.707717 7571 tablet.cc:2369] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:38:47.712405 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.710479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59792 (local address 127.2.95.66:41743)
0927 18:38:47.710666 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:47.710671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:47.710691 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:38:47.710831 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:47.710835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:47.710912 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:47.710994 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:47.711807 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:47.711813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:47.711817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:47.711833 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:47.711860 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:47.711872 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:47.711974 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:47.712115 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 18:38:47.712171 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:38:47.721438 7525 raft_consensus.cc:1261] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e [term 1 FOLLOWER]: Refusing update from remote peer 739c7a2eab454f1a8e6fe3487b48f488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:47.721437 7264 raft_consensus.cc:1261] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4 [term 1 FOLLOWER]: Refusing update from remote peer 739c7a2eab454f1a8e6fe3487b48f488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:38:47.722451 7599 consensus_queue.cc:1035] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92a6fdd29cef4d9a8e43abe157f990b4" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 34675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:47.722806 7594 consensus_queue.cc:1035] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae9c675628499c8fffb0d2c6646d7e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:38:47.750154 7123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:47.750418 7123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:38:47.754209 7123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff32bc7107264dc183c633d5501b3786: Sending DeleteTablet for 3 replicas of tablet 7cdddddfb417443db167f923b076e551
I20260927 18:38:47.755451 7244 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cdddddfb417443db167f923b076e551 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:47 UTC) from {username='slave'} at 127.0.0.1:59524
I20260927 18:38:47.755638 7505 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cdddddfb417443db167f923b076e551 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:47 UTC) from {username='slave'} at 127.0.0.1:44668
I20260927 18:38:47.755822 7375 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cdddddfb417443db167f923b076e551 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:38:47 UTC) from {username='slave'} at 127.0.0.1:59786
I20260927 18:38:47.757941 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 7182
I20260927 18:38:47.769326 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.758817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.2.95.65:34675)
0927 18:38:47.759005 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:47.759011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:47.765080 (+ 6069us) server_negotiation.cc:407] Connection header received
0927 18:38:47.766128 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:47.766135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:47.766253 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:47.766332 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:47.768368 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:47.768375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:47.768380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:47.768403 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:47.768446 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:47.768463 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:47.768568 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:47.768962 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 18:38:47.769030 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:38:47.770797 7304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:47.775177 7613 tablet_replica.cc:330] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e: stopping tablet replica
I20260927 18:38:47.775720 7613 raft_consensus.cc:2229] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:47.776278 7613 raft_consensus.cc:2258] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:47.777911 7614 tablet_replica.cc:330] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488: stopping tablet replica
I20260927 18:38:47.778383 7614 raft_consensus.cc:2229] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:38:47.779493 7614 raft_consensus.cc:2258] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:47.782361 7613 ts_tablet_manager.cc:1905] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:47.782372 7614 ts_tablet_manager.cc:1905] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:38:47.791267 7614 ts_tablet_manager.cc:1918] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:47.791498 7614 log.cc:1196] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/7cdddddfb417443db167f923b076e551
I20260927 18:38:47.793645 7614 ts_tablet_manager.cc:1939] T 7cdddddfb417443db167f923b076e551 P 739c7a2eab454f1a8e6fe3487b48f488: Deleting consensus metadata
I20260927 18:38:47.794323 7613 ts_tablet_manager.cc:1918] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:38:47.794564 7613 log.cc:1196] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/7cdddddfb417443db167f923b076e551
I20260927 18:38:47.795140 7613 ts_tablet_manager.cc:1939] T 7cdddddfb417443db167f923b076e551 P 07ae9c675628499c8fffb0d2c6646d7e: Deleting consensus metadata
I20260927 18:38:47.795639 7108 catalog_manager.cc:4841] TS 739c7a2eab454f1a8e6fe3487b48f488 (127.2.95.66:41743): tablet 7cdddddfb417443db167f923b076e551 (table test-table [id=189456d8bcba437fa84d6f165b26b740]) successfully deleted
I20260927 18:38:47.797044 7108 catalog_manager.cc:4841] TS 07ae9c675628499c8fffb0d2c6646d7e (127.2.95.67:45863): tablet 7cdddddfb417443db167f923b076e551 (table test-table [id=189456d8bcba437fa84d6f165b26b740]) successfully deleted
I20260927 18:38:47.883296 7612 tablet_replica.cc:330] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4: stopping tablet replica
I20260927 18:38:47.883754 7612 raft_consensus.cc:2229] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:38:47.884179 7612 raft_consensus.cc:2258] T 7cdddddfb417443db167f923b076e551 P 92a6fdd29cef4d9a8e43abe157f990b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:38:47.884547 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 7182
W20260927 18:38:47.898880 7108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.95.65:34675 (error 108)
I20260927 18:38:47.899120 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 7313
I20260927 18:38:47.900911 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.900468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34675 (local address 127.0.0.1:59534)
0927 18:38:47.900637 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:47.900769 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 18:38:47.901129 7108 catalog_manager.cc:4584] TS 92a6fdd29cef4d9a8e43abe157f990b4 (127.2.95.65:34675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7cdddddfb417443db167f923b076e551: Network error: Client connection negotiation failed: client connection to 127.2.95.65:34675: connect: Connection refused (error 111)
I20260927 18:38:47.902666 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.899986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.2.95.66:41743)
0927 18:38:47.900158 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:47.900163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:47.900184 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:38:47.900397 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:47.900401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:47.900453 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:47.900546 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:47.901978 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:47.901985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:47.901989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:47.902005 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:47.902043 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:47.902061 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:47.902191 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:47.902297 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 18:38:47.902359 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:38:47.903374 7435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:47.954964 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.954474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34675 (local address 127.0.0.1:59540)
0927 18:38:47.954669 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:47.954747 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 18:38:47.955897 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:47.955564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34675 (local address 127.0.0.1:59546)
0927 18:38:47.955705 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:47.955768 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 18:38:48.013473 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 7313
I20260927 18:38:48.016369 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:48.015922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34675 (local address 127.0.0.1:59558)
0927 18:38:48.016118 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:48.016201 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 18:38:48.017310 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:48.016972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34675 (local address 127.0.0.1:59562)
0927 18:38:48.017109 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:48.017183 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 18:38:48.028949 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 7443
I20260927 18:38:48.031539 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:48.029746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.2.95.67:45863)
0927 18:38:48.030004 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:48.030009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:48.030027 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:38:48.030084 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:48.030089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:48.030171 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:48.030257 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:48.031021 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:48.031025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:48.031028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:48.031042 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:48.031072 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:48.031083 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:48.031189 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:48.031220 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 18:38:48.031263 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:38:48.032147 7565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:48.098439 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:48.097886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34675 (local address 127.0.0.1:59566)
0927 18:38:48.098106 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:48.098218 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 18:38:48.099606 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:48.099179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:34675 (local address 127.0.0.1:59578)
0927 18:38:48.099410 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:48.099486 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 18:38:48.119235 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 7443
I20260927 18:38:48.133522 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 7092
I20260927 18:38:48.134743 7153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:48.237937 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 7092
2026-09-27T18:38:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2750 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 18:38:48.275969 2429 mini_oidc.cc:152] Starting JWKS server
I20260927 18:38:48.278585 2429 webserver.cc:460] Webserver started at https://127.0.0.1:38311/ using document root <none> and password file <none>
I20260927 18:38:48.278896 2429 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:38:48.279574 2429 webserver.cc:460] Webserver started at http://127.0.0.1:40323/ using document root <none> and password file <none>
2026-09-27T18:38:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:48Z Disabled control of system clock
I20260927 18:38:48.306092 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:41565
--webserver_interface=127.2.95.126
--webserver_port=0
--builtin_ntp_servers=127.2.95.84:43885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:41565
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38311/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/testchainca.cert with env {}
W20260927 18:38:48.630650 7626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:48.630972 7626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:48.631060 7626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:48.638464 7626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:48.638584 7626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:48.638619 7626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:48.638674 7626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:48.638716 7626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:38:48.638773 7626 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:38:48.638811 7626 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38311/jwks.json
I20260927 18:38:48.649477 7626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:43885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:41565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:41565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38311/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:48.650974 7626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:48.652369 7626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:48.665217 7631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:48.665263 7634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:48.665185 7632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:48.666266 7626 server_base.cc:1034] running on GCE node
I20260927 18:38:48.667286 7626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:48.668884 7626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:48.670161 7626 hybrid_clock.cc:648] HybridClock initialized: now 1790534328670111 us; error 47 us; skew 500 ppm
I20260927 18:38:48.670509 7626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:38:48.673082 7626 webserver.cc:460] Webserver started at http://127.2.95.126:33525/ using document root <none> and password file <none>
I20260927 18:38:48.673843 7626 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:48.673969 7626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:48.674268 7626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:48.677266 7626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "57d02da3201248d2975c0a0a2f99c446"
format_stamp: "Formatted at 2026-09-27 18:38:48 on dist-test-slave-9jll"
server_key: "20c937a22997a29b42e37d9e94ded862"
server_key_iv: "51b30aea798fb57b65b9985d006c0183"
server_key_version: "encryptionkey@0"
I20260927 18:38:48.678138 7626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "57d02da3201248d2975c0a0a2f99c446"
format_stamp: "Formatted at 2026-09-27 18:38:48 on dist-test-slave-9jll"
server_key: "20c937a22997a29b42e37d9e94ded862"
server_key_iv: "51b30aea798fb57b65b9985d006c0183"
server_key_version: "encryptionkey@0"
I20260927 18:38:48.683581 7626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 18:38:48.687392 7640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:48.688831 7626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:38:48.689026 7626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "57d02da3201248d2975c0a0a2f99c446"
format_stamp: "Formatted at 2026-09-27 18:38:48 on dist-test-slave-9jll"
server_key: "20c937a22997a29b42e37d9e94ded862"
server_key_iv: "51b30aea798fb57b65b9985d006c0183"
server_key_version: "encryptionkey@0"
I20260927 18:38:48.689236 7626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:48.714793 7626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:48.715952 7626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:48.716298 7626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:48.735831 7626 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:41565
I20260927 18:38:48.735853 7691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:41565 every 8 connection(s)
I20260927 18:38:48.737340 7626 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:48.741891 7692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:48.746407 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 7626
I20260927 18:38:48.746666 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:48.747380 2429 external_mini_cluster.cc:1428] Setting key 0ae31d8803bd88b168c957b4bef4f248
I20260927 18:38:48.754139 7692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446: Bootstrap starting.
I20260927 18:38:48.758455 7692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:48.759836 7692 log.cc:824] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:48.763149 7692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446: No bootstrap required, opened a new log
I20260927 18:38:48.771152 7692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d02da3201248d2975c0a0a2f99c446" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 41565 } }
I20260927 18:38:48.771613 7692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:48.771718 7692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d02da3201248d2975c0a0a2f99c446, State: Initialized, Role: FOLLOWER
I20260927 18:38:48.772486 7692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d02da3201248d2975c0a0a2f99c446" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 41565 } }
I20260927 18:38:48.772891 7692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:48.773059 7692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:48.773250 7692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:48.774108 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:48.749744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.2.95.126:41565)
0927 18:38:48.750954 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:48.750968 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:48.751021 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:38:48.752921 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:48.752951 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:48.753588 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:48.754058 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:48.754778 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:48.756311 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:48.758235 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:48.759101 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:48.759258 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:48.770795 (+ 11537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:48.770806 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:48.770827 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:48.771724 (+ 897us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:48.771857 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:48.771903 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:48.772040 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:48.772292 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 18:38:48.772708 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":470,"threads_started":1}
I20260927 18:38:48.776619 7692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d02da3201248d2975c0a0a2f99c446" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 41565 } }
I20260927 18:38:48.777212 7692 leader_election.cc:302] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57d02da3201248d2975c0a0a2f99c446; no voters:
I20260927 18:38:48.777956 7692 leader_election.cc:288] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:48.778127 7697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:48.779142 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:48.779191 7697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [term 1 LEADER]: Becoming Leader. State: Replica: 57d02da3201248d2975c0a0a2f99c446, State: Running, Role: LEADER
I20260927 18:38:48.779820 7697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d02da3201248d2975c0a0a2f99c446" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 41565 } }
I20260927 18:38:48.780458 7692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:48.782722 7699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57d02da3201248d2975c0a0a2f99c446. Latest consensus state: current_term: 1 leader_uuid: "57d02da3201248d2975c0a0a2f99c446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d02da3201248d2975c0a0a2f99c446" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 41565 } } }
I20260927 18:38:48.783196 7699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:48.783681 7698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57d02da3201248d2975c0a0a2f99c446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d02da3201248d2975c0a0a2f99c446" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 41565 } } }
I20260927 18:38:48.784000 7698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:48.785710 7703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:48.792085 7703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:48.804829 7703 catalog_manager.cc:1347] Generated new cluster ID: 8c2e7878baa047939d94b8cde82a2a14
I20260927 18:38:48.804989 7703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:48.830775 7703 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:48.832633 7703 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:48.848012 7703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57d02da3201248d2975c0a0a2f99c446: Generated new TSK 0
I20260927 18:38:48.849159 7703 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 18:38:48.901733 7722 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 18:38:48.902004 7722 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:38:48.902837 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:48.872415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.2.95.126:41565)
0927 18:38:48.872725 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:48.872732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:48.873199 (+ 467us) server_negotiation.cc:407] Connection header received
0927 18:38:48.873559 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:48.873581 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:48.873691 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:48.873840 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:38:48.874496 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:48.875321 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:48.877478 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:48.878050 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:48.878157 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:48.878894 (+ 737us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:38:48.902207 (+ 23313us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 18:38:48.902584 (+ 377us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58148: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 18:38:48.902961 7695 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58148: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 18:38:48.905128 7721 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:48.872290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:41565 (local address 127.0.0.1:58148)
0927 18:38:48.873075 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:48.873097 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:48.873317 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:48.873883 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:48.873889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:48.873904 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:38:48.874309 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:48.874322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:48.875462 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:48.875469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:48.877314 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:48.877330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:48.878243 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:48.878251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:48.878449 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:48.878550 (+ 101us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:38:48.902559 (+ 24009us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 18:38:48.902995 (+ 436us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.95.126:41565: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":409,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.95.126:41565: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 18:38:48.906721 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 7626
I20260927 18:38:48.907985 7687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:49.013556 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 7626
2026-09-27T18:38:49Z chronyd exiting
I20260927 18:38:49.064518 2429 test_util.cc:175] -----------------------------------------------
I20260927 18:38:49.064713 2429 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790534284715420-2429-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (793 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 18:38:49.069581 2429 mini_oidc.cc:152] Starting JWKS server
I20260927 18:38:49.071780 2429 webserver.cc:460] Webserver started at https://127.0.0.1:41307/ using document root <none> and password file <none>
I20260927 18:38:49.072024 2429 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:38:49.072827 2429 webserver.cc:460] Webserver started at http://127.0.0.1:43645/ using document root <none> and password file <none>
2026-09-27T18:38:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:49Z Disabled control of system clock
I20260927 18:38:49.104555 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33685
--webserver_interface=127.2.95.126
--webserver_port=0
--builtin_ntp_servers=127.2.95.84:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:33685
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41307/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/testchainca.cert with env {}
W20260927 18:38:49.478603 7729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:49.478986 7729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:49.479113 7729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:49.486879 7729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:49.487039 7729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:49.487106 7729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:49.487195 7729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:49.487258 7729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:38:49.487352 7729 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:38:49.487411 7729 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41307/jwks.json
I20260927 18:38:49.498543 7729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:33685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41307/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:49.500334 7729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:49.501971 7729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:49.515252 7734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:49.515198 7737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:49.515220 7735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:49.515898 7729 server_base.cc:1034] running on GCE node
I20260927 18:38:49.516960 7729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:49.518826 7729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:49.520143 7729 hybrid_clock.cc:648] HybridClock initialized: now 1790534329519999 us; error 53 us; skew 500 ppm
I20260927 18:38:49.520551 7729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:38:49.523365 7729 webserver.cc:460] Webserver started at http://127.2.95.126:33605/ using document root <none> and password file <none>
I20260927 18:38:49.524209 7729 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:49.524307 7729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:49.524672 7729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:49.528082 7729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "583184a1eeac42cbb58bd5331f3a01b6"
format_stamp: "Formatted at 2026-09-27 18:38:49 on dist-test-slave-9jll"
server_key: "9462355d6e279f885b7125ab1206a7e8"
server_key_iv: "e85b2c0757aa767ef47538888508f473"
server_key_version: "encryptionkey@0"
I20260927 18:38:49.528959 7729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "583184a1eeac42cbb58bd5331f3a01b6"
format_stamp: "Formatted at 2026-09-27 18:38:49 on dist-test-slave-9jll"
server_key: "9462355d6e279f885b7125ab1206a7e8"
server_key_iv: "e85b2c0757aa767ef47538888508f473"
server_key_version: "encryptionkey@0"
I20260927 18:38:49.535151 7729 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 18:38:49.539403 7743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:49.541115 7729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:38:49.541316 7729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "583184a1eeac42cbb58bd5331f3a01b6"
format_stamp: "Formatted at 2026-09-27 18:38:49 on dist-test-slave-9jll"
server_key: "9462355d6e279f885b7125ab1206a7e8"
server_key_iv: "e85b2c0757aa767ef47538888508f473"
server_key_version: "encryptionkey@0"
I20260927 18:38:49.541555 7729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:49.556174 7729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:49.557201 7729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:49.557493 7729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:49.578529 7729 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:33685
I20260927 18:38:49.578550 7794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:33685 every 8 connection(s)
I20260927 18:38:49.580173 7729 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:49.584909 7795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:49.588418 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 7729
I20260927 18:38:49.588629 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:49.589704 2429 external_mini_cluster.cc:1428] Setting key be481f77440db5a2715b0f81382c8dc2
I20260927 18:38:49.599223 7795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6: Bootstrap starting.
I20260927 18:38:49.603008 7795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:49.604272 7795 log.cc:824] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:49.604813 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:49.592283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.2.95.126:33685)
0927 18:38:49.593493 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:49.593507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:49.593560 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:38:49.595470 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:49.595497 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:49.596120 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:49.596585 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:49.597238 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:49.598639 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:49.599996 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:49.600710 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:49.600899 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:49.602239 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:49.602248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:49.602264 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:49.602865 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:49.602968 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:49.602990 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:49.603106 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:49.603338 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 18:38:49.603680 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":402,"threads_started":1}
I20260927 18:38:49.607872 7795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6: No bootstrap required, opened a new log
I20260927 18:38:49.609380 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:49.616254 7795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "583184a1eeac42cbb58bd5331f3a01b6" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33685 } }
I20260927 18:38:49.616712 7795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:49.616809 7795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583184a1eeac42cbb58bd5331f3a01b6, State: Initialized, Role: FOLLOWER
I20260927 18:38:49.617466 7795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "583184a1eeac42cbb58bd5331f3a01b6" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33685 } }
I20260927 18:38:49.617750 7795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:49.617889 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:49.618059 7795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:49.620675 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "583184a1eeac42cbb58bd5331f3a01b6" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33685 } }
I20260927 18:38:49.621275 7795 leader_election.cc:302] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 583184a1eeac42cbb58bd5331f3a01b6; no voters:
I20260927 18:38:49.622004 7795 leader_election.cc:288] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:49.622200 7800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:49.622880 7800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [term 1 LEADER]: Becoming Leader. State: Replica: 583184a1eeac42cbb58bd5331f3a01b6, State: Running, Role: LEADER
I20260927 18:38:49.623454 7800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "583184a1eeac42cbb58bd5331f3a01b6" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33685 } }
I20260927 18:38:49.624320 7795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:49.626518 7802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 583184a1eeac42cbb58bd5331f3a01b6. Latest consensus state: current_term: 1 leader_uuid: "583184a1eeac42cbb58bd5331f3a01b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "583184a1eeac42cbb58bd5331f3a01b6" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33685 } } }
I20260927 18:38:49.627331 7802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:49.626847 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "583184a1eeac42cbb58bd5331f3a01b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "583184a1eeac42cbb58bd5331f3a01b6" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33685 } } }
I20260927 18:38:49.627930 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:49.628414 7809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:49.634402 7809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:49.645610 7809 catalog_manager.cc:1347] Generated new cluster ID: b222a7efc7a141e6a1885b45ac57d82c
I20260927 18:38:49.645810 7809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:49.668916 7809 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:49.670629 7809 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:49.680209 7809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 583184a1eeac42cbb58bd5331f3a01b6: Generated new TSK 0
I20260927 18:38:49.681303 7809 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 18:38:49.731228 7825 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 18:38:49.731389 7825 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:38:49.732697 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:49.705132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.2.95.126:33685)
0927 18:38:49.705382 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:49.705391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:49.705775 (+ 384us) server_negotiation.cc:407] Connection header received
0927 18:38:49.706076 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:49.706081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:49.706168 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:49.706324 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:38:49.706838 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:49.707626 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:49.709006 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:49.709489 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:49.709607 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:49.710155 (+ 548us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:38:49.731867 (+ 21712us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 18:38:49.732343 (+ 476us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48040: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":100}
W20260927 18:38:49.732851 7798 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48040: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 18:38:49.733098 7824 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:49.705039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:33685 (local address 127.0.0.1:48040)
0927 18:38:49.705638 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:49.705655 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:49.705848 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:49.706359 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:49.706364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:49.706388 (+ 24us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:38:49.706678 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:49.706694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:49.707796 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:49.707800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:49.708870 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:49.708882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:49.709656 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:49.709684 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:49.709868 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:49.709905 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:38:49.732322 (+ 22417us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 18:38:49.732632 (+ 310us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.95.126:33685: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":339,"threads_started":1}
I20260927 18:38:49.735114 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 7729
I20260927 18:38:49.736675 7790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:49.833377 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 7729
2026-09-27T18:38:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (819 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 18:38:49.888099 2429 mini_oidc.cc:152] Starting JWKS server
I20260927 18:38:49.890173 2429 webserver.cc:460] Webserver started at https://127.0.0.1:46783/ using document root <none> and password file <none>
I20260927 18:38:49.890415 2429 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 18:38:49.891047 2429 webserver.cc:460] Webserver started at http://127.0.0.1:33633/ using document root <none> and password file <none>
2026-09-27T18:38:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:49Z Disabled control of system clock
I20260927 18:38:49.922405 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:37029
--webserver_interface=127.2.95.126
--webserver_port=0
--builtin_ntp_servers=127.2.95.84:41911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:37029
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46783/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 18:38:50.279462 7832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:50.279763 7832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:50.279847 7832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:50.287150 7832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:50.287274 7832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:50.287310 7832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:50.287360 7832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:50.287406 7832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:38:50.287458 7832 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:38:50.287495 7832 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46783/jwks.json
I20260927 18:38:50.296974 7832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:37029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:37029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46783/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:50.298288 7832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:50.300087 7832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:50.311677 7837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:50.311616 7838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:50.311656 7840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:50.313050 7832 server_base.cc:1034] running on GCE node
I20260927 18:38:50.313948 7832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:50.315497 7832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:50.316733 7832 hybrid_clock.cc:648] HybridClock initialized: now 1790534330316672 us; error 47 us; skew 500 ppm
I20260927 18:38:50.317119 7832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:38:50.319689 7832 webserver.cc:460] Webserver started at http://127.2.95.126:44817/ using document root <none> and password file <none>
I20260927 18:38:50.320444 7832 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:50.320554 7832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:50.320896 7832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:50.325119 7832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "5268452de63448b0892839d638c030b7"
format_stamp: "Formatted at 2026-09-27 18:38:50 on dist-test-slave-9jll"
server_key: "d24829b3f070ee9c2c1f96cbb38af1ee"
server_key_iv: "9f46f5c79d1bd81e2da7b68946b7dd79"
server_key_version: "encryptionkey@0"
I20260927 18:38:50.326282 7832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "5268452de63448b0892839d638c030b7"
format_stamp: "Formatted at 2026-09-27 18:38:50 on dist-test-slave-9jll"
server_key: "d24829b3f070ee9c2c1f96cbb38af1ee"
server_key_iv: "9f46f5c79d1bd81e2da7b68946b7dd79"
server_key_version: "encryptionkey@0"
I20260927 18:38:50.333061 7832 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260927 18:38:50.337193 7846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:50.338848 7832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:38:50.339116 7832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "5268452de63448b0892839d638c030b7"
format_stamp: "Formatted at 2026-09-27 18:38:50 on dist-test-slave-9jll"
server_key: "d24829b3f070ee9c2c1f96cbb38af1ee"
server_key_iv: "9f46f5c79d1bd81e2da7b68946b7dd79"
server_key_version: "encryptionkey@0"
I20260927 18:38:50.339332 7832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:50.358428 7832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:50.359319 7832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:50.359568 7832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:50.377382 7832 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:37029
I20260927 18:38:50.377400 7897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:37029 every 8 connection(s)
I20260927 18:38:50.378757 7832 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:50.383036 7898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:50.384132 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 7832
I20260927 18:38:50.384330 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:50.384925 2429 external_mini_cluster.cc:1428] Setting key f8620399da5ac4b60635bce199a0dbc4
I20260927 18:38:50.396380 7898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7: Bootstrap starting.
I20260927 18:38:50.398872 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:50.387033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.2.95.126:37029)
0927 18:38:50.388205 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:50.388217 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:50.388262 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:38:50.389894 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:50.389924 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:50.390499 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:50.390875 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:50.391491 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:50.392802 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:50.394199 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:50.394902 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:50.395072 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:50.396487 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:50.396495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:50.396511 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:50.397081 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:50.397163 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:50.397185 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:50.397292 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:50.397528 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 18:38:50.397893 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":422,"threads_started":1}
I20260927 18:38:50.399729 7898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:50.400815 7898 log.cc:824] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:50.402761 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:50.403949 7898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7: No bootstrap required, opened a new log
I20260927 18:38:50.412022 7898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5268452de63448b0892839d638c030b7" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37029 } }
I20260927 18:38:50.412426 7898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:50.412515 7898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5268452de63448b0892839d638c030b7, State: Initialized, Role: FOLLOWER
I20260927 18:38:50.413074 7898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5268452de63448b0892839d638c030b7" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37029 } }
I20260927 18:38:50.413287 7898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:50.413405 7898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:50.413554 7898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:50.415704 7898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5268452de63448b0892839d638c030b7" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37029 } }
I20260927 18:38:50.416213 7898 leader_election.cc:302] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5268452de63448b0892839d638c030b7; no voters:
I20260927 18:38:50.416771 7898 leader_election.cc:288] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:50.416922 7903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:50.417646 7903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [term 1 LEADER]: Becoming Leader. State: Replica: 5268452de63448b0892839d638c030b7, State: Running, Role: LEADER
I20260927 18:38:50.418177 7903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5268452de63448b0892839d638c030b7" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37029 } }
I20260927 18:38:50.418694 7898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:50.421252 7905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5268452de63448b0892839d638c030b7. Latest consensus state: current_term: 1 leader_uuid: "5268452de63448b0892839d638c030b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5268452de63448b0892839d638c030b7" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37029 } } }
I20260927 18:38:50.421703 7905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:50.421723 7904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5268452de63448b0892839d638c030b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5268452de63448b0892839d638c030b7" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37029 } } }
I20260927 18:38:50.422315 7904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:50.424016 7912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:50.428844 7912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:50.437836 7912 catalog_manager.cc:1347] Generated new cluster ID: 4fa42bfc24164544876249889fb106e6
I20260927 18:38:50.437965 7912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:50.458468 7912 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:50.459923 7912 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:50.472345 7912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5268452de63448b0892839d638c030b7: Generated new TSK 0
I20260927 18:38:50.473330 7912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:50.502409 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:50.493211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.2.95.126:37029)
0927 18:38:50.493434 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:50.493440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:50.493789 (+ 349us) server_negotiation.cc:407] Connection header received
0927 18:38:50.494105 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:50.494111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:50.494214 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:50.494402 (+ 188us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:38:50.494844 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:50.495991 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:50.497704 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:50.498428 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:50.498585 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:50.499178 (+ 593us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:38:50.501701 (+ 2523us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 18:38:50.502102 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 18:38:50.502549 7901 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 18:38:50.504108 7927 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:50.493037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37029 (local address 127.0.0.1:49718)
0927 18:38:50.493660 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:50.493701 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:50.493911 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:50.494425 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:50.494428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:50.494439 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:38:50.494696 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:50.494704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:50.496185 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:50.496191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:50.497435 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:50.497453 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:50.498674 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:50.498680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:50.498826 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:50.498880 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:38:50.502027 (+ 3147us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 18:38:50.503656 (+ 1629us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.95.126:37029: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46783/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":1289,"thread_start_us":345,"threads_started":1}
I20260927 18:38:50.506201 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 7832
I20260927 18:38:50.507576 7893 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 18:38:50.514012 7928 webserver.cc:533] Webserver: sslize failed:
E20260927 18:38:50.514113 7928 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 18:38:50.599460 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 7832
2026-09-27T18:38:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (763 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T18:38:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:50Z Disabled control of system clock
I20260927 18:38:50.678156 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:45387
--webserver_interface=127.2.95.126
--webserver_port=0
--builtin_ntp_servers=127.2.95.84:45191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:45387
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/testchainca.cert with env {}
W20260927 18:38:51.004094 7933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:51.004426 7933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:51.004513 7933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:51.011866 7933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:51.011996 7933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:51.012032 7933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:51.012084 7933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:51.012126 7933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 18:38:51.012184 7933 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 18:38:51.012223 7933 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 18:38:51.022514 7933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:45191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:45387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:45387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:51.023818 7933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:51.025159 7933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:51.037639 7938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:51.037599 7941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:51.037698 7939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:51.038656 7933 server_base.cc:1034] running on GCE node
I20260927 18:38:51.039549 7933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:51.040964 7933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:51.042161 7933 hybrid_clock.cc:648] HybridClock initialized: now 1790534331042117 us; error 36 us; skew 500 ppm
I20260927 18:38:51.042505 7933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:38:51.044816 7933 webserver.cc:460] Webserver started at http://127.2.95.126:41771/ using document root <none> and password file <none>
I20260927 18:38:51.045511 7933 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:51.045610 7933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:51.045936 7933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:51.048873 7933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "c6b26ea529ca4191ab00abff9749e173"
format_stamp: "Formatted at 2026-09-27 18:38:51 on dist-test-slave-9jll"
server_key: "ce2e3fa5765e871ba5eb479f4f470d0d"
server_key_iv: "f404e02368075906b3466e6469232c28"
server_key_version: "encryptionkey@0"
I20260927 18:38:51.049705 7933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "c6b26ea529ca4191ab00abff9749e173"
format_stamp: "Formatted at 2026-09-27 18:38:51 on dist-test-slave-9jll"
server_key: "ce2e3fa5765e871ba5eb479f4f470d0d"
server_key_iv: "f404e02368075906b3466e6469232c28"
server_key_version: "encryptionkey@0"
I20260927 18:38:51.054986 7933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:38:51.058713 7947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:51.060180 7933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:38:51.060359 7933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "c6b26ea529ca4191ab00abff9749e173"
format_stamp: "Formatted at 2026-09-27 18:38:51 on dist-test-slave-9jll"
server_key: "ce2e3fa5765e871ba5eb479f4f470d0d"
server_key_iv: "f404e02368075906b3466e6469232c28"
server_key_version: "encryptionkey@0"
I20260927 18:38:51.060578 7933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:51.080281 7933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:51.081311 7933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:51.081601 7933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:51.099536 7933 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:45387
I20260927 18:38:51.099556 7998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:45387 every 8 connection(s)
I20260927 18:38:51.100828 7933 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:51.104681 7999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:51.108170 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 7933
I20260927 18:38:51.108368 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:51.109157 2429 external_mini_cluster.cc:1428] Setting key e404158f5c74ad318fc16db5656d2727
I20260927 18:38:51.117256 7999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173: Bootstrap starting.
I20260927 18:38:51.120357 7999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:51.121492 7999 log.cc:824] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:51.122208 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:51.111135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.2.95.126:45387)
0927 18:38:51.112078 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:51.112090 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:51.112129 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:38:51.113764 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:51.113788 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:51.114296 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:51.114714 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:51.115354 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:51.116614 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:51.117958 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:51.118614 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:51.118786 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:51.119864 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:51.119871 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:51.119886 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:38:51.120395 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:51.120498 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:38:51.120524 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:51.120624 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:51.120833 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 18:38:51.121135 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":340,"threads_started":1}
I20260927 18:38:51.124444 7999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173: No bootstrap required, opened a new log
I20260927 18:38:51.125852 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:38:51.132462 7999 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b26ea529ca4191ab00abff9749e173" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45387 } }
I20260927 18:38:51.132959 7999 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:51.133078 7999 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b26ea529ca4191ab00abff9749e173, State: Initialized, Role: FOLLOWER
I20260927 18:38:51.133771 7999 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b26ea529ca4191ab00abff9749e173" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45387 } }
I20260927 18:38:51.134029 7999 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:51.134158 7999 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:51.134297 7999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:51.136425 7999 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b26ea529ca4191ab00abff9749e173" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45387 } }
I20260927 18:38:51.136969 7999 leader_election.cc:302] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6b26ea529ca4191ab00abff9749e173; no voters:
I20260927 18:38:51.137527 7999 leader_election.cc:288] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:51.137713 8004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:51.138859 8004 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [term 1 LEADER]: Becoming Leader. State: Replica: c6b26ea529ca4191ab00abff9749e173, State: Running, Role: LEADER
I20260927 18:38:51.139475 8004 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b26ea529ca4191ab00abff9749e173" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45387 } }
I20260927 18:38:51.139792 7999 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:51.142123 8006 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b26ea529ca4191ab00abff9749e173. Latest consensus state: current_term: 1 leader_uuid: "c6b26ea529ca4191ab00abff9749e173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b26ea529ca4191ab00abff9749e173" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45387 } } }
I20260927 18:38:51.142107 8005 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6b26ea529ca4191ab00abff9749e173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b26ea529ca4191ab00abff9749e173" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45387 } } }
I20260927 18:38:51.142602 8005 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:51.142602 8006 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:51.145287 8011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:51.150231 8011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:51.159864 8011 catalog_manager.cc:1347] Generated new cluster ID: 7adf0c7ed4e043479d12f4d122dddadc
I20260927 18:38:51.159996 8011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:51.171353 8011 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:51.173260 8011 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:51.181113 8011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6b26ea529ca4191ab00abff9749e173: Generated new TSK 0
I20260927 18:38:51.182266 8011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:38:51.776331 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:51.196752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.2.95.126:45387)
0927 18:38:51.196947 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:51.196957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:51.197450 (+ 493us) server_negotiation.cc:407] Connection header received
0927 18:38:51.197606 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:51.197611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:51.197743 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:51.197944 (+ 201us) server_negotiation.cc:226] Negotiated authn=JWT
0927 18:38:51.198401 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:51.199539 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:51.200868 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:51.201580 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:51.201750 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:51.202190 (+ 440us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 18:38:51.775776 (+573586us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 18:38:51.776107 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54108: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 18:38:51.776432 8002 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54108: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 18:38:51.776998 8028 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:51.196683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45387 (local address 127.0.0.1:54108)
0927 18:38:51.197307 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:51.197323 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:51.197445 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:51.197939 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:51.197942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:51.197952 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 18:38:51.198252 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:51.198261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:51.199671 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:51.199675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:51.200733 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:51.200744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:51.201760 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:51.201764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:51.201882 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:51.201919 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 18:38:51.776065 (+574146us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 18:38:51.776315 (+ 250us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.95.126:45387: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":358,"threads_started":1}
I20260927 18:38:51.778579 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 7933
I20260927 18:38:51.779712 7994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:51.881384 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 7933
2026-09-27T18:38:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1262 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (412 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1417 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:53 dist-test-slave-9jll krb5kdc[8044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:53 dist-test-slave-9jll krb5kdc[8044](info): set up 2 sockets
Sep 27 18:38:53 dist-test-slave-9jll krb5kdc[8044](info): commencing operation
krb5kdc: starting...
W20260927 18:38:54.730536 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:38:54 dist-test-slave-9jll krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:38:54.941880 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:39711
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:39711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:55.271371 8057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:55.271690 8057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:55.271826 8057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:55.279270 8057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:55.279412 8057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:55.279491 8057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:55.279563 8057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:55.279620 8057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:55.289600 8057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:39711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:55.291034 8057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:55.292330 8057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:55.304129 8062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:55.304111 8063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:55.304104 8065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:55.305348 8057 server_base.cc:1034] running on GCE node
I20260927 18:38:55.306308 8057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:55.307775 8057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:55.309036 8057 hybrid_clock.cc:648] HybridClock initialized: now 1790534335308978 us; error 48 us; skew 500 ppm
I20260927 18:38:55.309417 8057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:55 dist-test-slave-9jll krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:55.312619 8057 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:38:55.314214 8057 webserver.cc:460] Webserver started at http://127.2.95.126:43185/ using document root <none> and password file <none>
I20260927 18:38:55.314917 8057 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:55.315021 8057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:55.315339 8057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:55.318354 8057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "86566759e74440569b16275066a7bc50"
format_stamp: "Formatted at 2026-09-27 18:38:55 on dist-test-slave-9jll"
server_key: "ceb293b71250757259bce310e8ab2503"
server_key_iv: "cbec6668ceef0536379a195d2c4d59eb"
server_key_version: "encryptionkey@0"
I20260927 18:38:55.319116 8057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "86566759e74440569b16275066a7bc50"
format_stamp: "Formatted at 2026-09-27 18:38:55 on dist-test-slave-9jll"
server_key: "ceb293b71250757259bce310e8ab2503"
server_key_iv: "cbec6668ceef0536379a195d2c4d59eb"
server_key_version: "encryptionkey@0"
I20260927 18:38:55.324435 8057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:38:55.328210 8072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:55.329859 8057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:38:55.330050 8057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "86566759e74440569b16275066a7bc50"
format_stamp: "Formatted at 2026-09-27 18:38:55 on dist-test-slave-9jll"
server_key: "ceb293b71250757259bce310e8ab2503"
server_key_iv: "cbec6668ceef0536379a195d2c4d59eb"
server_key_version: "encryptionkey@0"
I20260927 18:38:55.330258 8057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:55.352773 8057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:55.357573 8057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:55.357897 8057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:55.376187 8057 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:39711
I20260927 18:38:55.376276 8123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:39711 every 8 connection(s)
I20260927 18:38:55.377602 8057 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:55.381496 8124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:55.383143 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 8057
I20260927 18:38:55.383356 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:55.384160 2429 external_mini_cluster.cc:1428] Setting key e498b99d387a5f587396c93ac2810f29
I20260927 18:38:55.394779 8124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50: Bootstrap starting.
I20260927 18:38:55.398830 8124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50: Neither blocks nor log segments found. Creating new log.
Sep 27 18:38:55 dist-test-slave-9jll krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:55.400084 8124 log.cc:824] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50: Log is configured to *not* fsync() on all Append() calls
I20260927 18:38:55.403439 8124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50: No bootstrap required, opened a new log
I20260927 18:38:55.409334 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:55.386335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.2.95.126:39711)
0927 18:38:55.387234 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:55.387257 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:55.387303 (+ 46us) server_negotiation.cc:407] Connection header received
0927 18:38:55.389088 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:55.389113 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:55.389534 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:55.389991 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:55.392048 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:55.393531 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:55.395387 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:55.396114 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:55.396264 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:55.401398 (+ 5134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:55.401411 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:55.401436 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:55.401480 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:55.404724 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:55.405717 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:55.405727 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:55.405734 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:55.405951 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:55.406381 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:55.406385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:55.406387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:55.407046 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:55.407277 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:55.407766 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 18:38:55.408209 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":397,"threads_started":1}
I20260927 18:38:55.412137 8124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86566759e74440569b16275066a7bc50" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39711 } }
I20260927 18:38:55.412559 8124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:55.412667 8124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86566759e74440569b16275066a7bc50, State: Initialized, Role: FOLLOWER
I20260927 18:38:55.413240 8124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86566759e74440569b16275066a7bc50" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39711 } }
I20260927 18:38:55.413479 8124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:55.413602 8124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:55.413797 8124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:55.415951 8124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86566759e74440569b16275066a7bc50" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39711 } }
I20260927 18:38:55.416491 8124 leader_election.cc:302] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86566759e74440569b16275066a7bc50; no voters:
I20260927 18:38:55.417095 8124 leader_election.cc:288] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:55.417246 8129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:55.417928 8129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [term 1 LEADER]: Becoming Leader. State: Replica: 86566759e74440569b16275066a7bc50, State: Running, Role: LEADER
I20260927 18:38:55.418689 8129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86566759e74440569b16275066a7bc50" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39711 } }
I20260927 18:38:55.419445 8124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:55.421871 8130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86566759e74440569b16275066a7bc50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86566759e74440569b16275066a7bc50" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39711 } } }
I20260927 18:38:55.421907 8131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86566759e74440569b16275066a7bc50. Latest consensus state: current_term: 1 leader_uuid: "86566759e74440569b16275066a7bc50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86566759e74440569b16275066a7bc50" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39711 } } }
I20260927 18:38:55.422365 8130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:55.422365 8131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50 [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:55.423261 8137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:55.428434 8137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:55.440059 8137 catalog_manager.cc:1347] Generated new cluster ID: 9ed4ec00f7734b329a2743e7d0bcc4b2
I20260927 18:38:55.440181 8137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:55.468978 8137 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:55.470407 8137 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:55.479544 8137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86566759e74440569b16275066a7bc50: Generated new TSK 0
I20260927 18:38:55.480458 8137 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:38:55.542963 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39711
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:55.880434 8150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:55.880731 8150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:55.880841 8150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:55.888227 8150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:55.888357 8150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:55.888520 8150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:38:55.898813 8150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:55.900228 8150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:55.901577 8150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:55.916101 8156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:55.916054 8155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:55.916136 8158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:55.916824 8150 server_base.cc:1034] running on GCE node
I20260927 18:38:55.917462 8150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:55.918514 8150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:55.919745 8150 hybrid_clock.cc:648] HybridClock initialized: now 1790534335919687 us; error 49 us; skew 500 ppm
I20260927 18:38:55.920095 8150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:55 dist-test-slave-9jll krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:55.923434 8150 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:38:55.925154 8150 webserver.cc:460] Webserver started at http://127.2.95.65:44029/ using document root <none> and password file <none>
I20260927 18:38:55.925976 8150 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:55.926092 8150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:55.926394 8150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:55.929782 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "938c6bb2040c46f08c287e31c4c0334d"
format_stamp: "Formatted at 2026-09-27 18:38:55 on dist-test-slave-9jll"
server_key: "bc024dfb7721bdf5f67bc08f30f3c729"
server_key_iv: "66319f3a6ecfd88e27b58e1739bc6eee"
server_key_version: "encryptionkey@0"
I20260927 18:38:55.930719 8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "938c6bb2040c46f08c287e31c4c0334d"
format_stamp: "Formatted at 2026-09-27 18:38:55 on dist-test-slave-9jll"
server_key: "bc024dfb7721bdf5f67bc08f30f3c729"
server_key_iv: "66319f3a6ecfd88e27b58e1739bc6eee"
server_key_version: "encryptionkey@0"
I20260927 18:38:55.936782 8150 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:38:55.941205 8165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:55.943006 8150 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 18:38:55.943198 8150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "938c6bb2040c46f08c287e31c4c0334d"
format_stamp: "Formatted at 2026-09-27 18:38:55 on dist-test-slave-9jll"
server_key: "bc024dfb7721bdf5f67bc08f30f3c729"
server_key_iv: "66319f3a6ecfd88e27b58e1739bc6eee"
server_key_version: "encryptionkey@0"
I20260927 18:38:55.943416 8150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:55.958529 8150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:55.963340 8150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:55.963644 8150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:55.964679 8150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:55.966329 8150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:55.966435 8150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:55.966563 8150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:55.966625 8150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:55.997440 8150 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:38483
I20260927 18:38:55.997486 8277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:38483 every 8 connection(s)
I20260927 18:38:55.998802 8150 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:56.006410 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 8150
I20260927 18:38:56.006625 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:56.007416 2429 external_mini_cluster.cc:1428] Setting key 962867d15d0b97dfdc51eaa51ad9ed03
Sep 27 18:38:56 dist-test-slave-9jll krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534335, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:56.023813 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:56.001940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:53203 (local address 127.2.95.126:39711)
0927 18:38:56.002455 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:56.002460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:56.003444 (+ 984us) server_negotiation.cc:407] Connection header received
0927 18:38:56.005451 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:56.005457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:56.005568 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:56.005703 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:56.007567 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:56.008376 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:56.010573 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:56.011149 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:56.011315 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:56.016473 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:56.016479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:56.016482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:56.016523 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:56.019348 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:56.020717 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:56.020723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:56.020726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:56.020949 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:56.021661 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:56.021721 (+ 60us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:56.021724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:56.022135 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:56.022290 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:56.023294 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0927 18:38:56.023537 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":256,"threads_started":1}
I20260927 18:38:56.025359 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:56.002385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39711 (local address 127.2.95.65:53203)
0927 18:38:56.003252 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:56.003295 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:56.005085 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:56.006030 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:56.006045 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:56.006680 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:56.007358 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:56.007379 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:56.008565 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:56.008571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:56.010397 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:56.010413 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:56.011445 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:56.011451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:56.011677 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:56.012472 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:56.012499 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:56.016083 (+ 3584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:56.019647 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:56.019659 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:56.019681 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:56.020407 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:56.021081 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:56.021086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:56.021089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:56.021379 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:56.022282 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:56.022291 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:56.022817 (+ 526us) client_negotiation.cc:769] Sending connection context
0927 18:38:56.023135 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 18:38:56.023760 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":330,"threads_started":1}
I20260927 18:38:56.027269 8278 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39711
I20260927 18:38:56.027644 8278 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:56.028635 8278 heartbeater.cc:511] Master 127.2.95.126:39711 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
I20260927 18:38:56.031144 8089 ts_manager.cc:194] Registered new tserver with Master: 938c6bb2040c46f08c287e31c4c0334d (127.2.95.65:38483)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
I20260927 18:38:56.032799 8089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:53203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:38:56.057910 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39711
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:56.389876 8284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:56.390270 8284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:56.390414 8284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:56.400543 8284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:56.400672 8284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:56.400789 8284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:38:56.411101 8284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:56.412355 8284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:56.413640 8284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:56.426972 8289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:56.426990 8292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:56.426928 8290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:56.427649 8284 server_base.cc:1034] running on GCE node
I20260927 18:38:56.428224 8284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:56.429167 8284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:56.430404 8284 hybrid_clock.cc:648] HybridClock initialized: now 1790534336430333 us; error 56 us; skew 500 ppm
I20260927 18:38:56.430755 8284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:56 dist-test-slave-9jll krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534336, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:56.434170 8284 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:38:56.435813 8284 webserver.cc:460] Webserver started at http://127.2.95.66:45941/ using document root <none> and password file <none>
I20260927 18:38:56.436491 8284 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:56.436601 8284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:56.436900 8284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:56.440068 8284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "59c3d7f89b8e470a9feb85e0b0f0ea69"
format_stamp: "Formatted at 2026-09-27 18:38:56 on dist-test-slave-9jll"
server_key: "a1075f2c260d6594a71b30427015b72a"
server_key_iv: "505bed2d2704ecaecd21bf6abdc891bc"
server_key_version: "encryptionkey@0"
I20260927 18:38:56.440979 8284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "59c3d7f89b8e470a9feb85e0b0f0ea69"
format_stamp: "Formatted at 2026-09-27 18:38:56 on dist-test-slave-9jll"
server_key: "a1075f2c260d6594a71b30427015b72a"
server_key_iv: "505bed2d2704ecaecd21bf6abdc891bc"
server_key_version: "encryptionkey@0"
I20260927 18:38:56.446607 8284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:38:56.450534 8299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:56.452025 8284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:38:56.452194 8284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "59c3d7f89b8e470a9feb85e0b0f0ea69"
format_stamp: "Formatted at 2026-09-27 18:38:56 on dist-test-slave-9jll"
server_key: "a1075f2c260d6594a71b30427015b72a"
server_key_iv: "505bed2d2704ecaecd21bf6abdc891bc"
server_key_version: "encryptionkey@0"
I20260927 18:38:56.452405 8284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:56.468916 8284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:56.473183 8284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:56.473462 8284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:56.474489 8284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:56.475966 8284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:56.476040 8284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:56.476147 8284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:56.476205 8284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:56.507517 8284 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:36011
I20260927 18:38:56.507566 8411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:36011 every 8 connection(s)
I20260927 18:38:56.508783 8284 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:38:56.519204 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 8284
I20260927 18:38:56.519377 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:38:56.519826 2429 external_mini_cluster.cc:1428] Setting key 8b2d75060c274fbe8d311a685a3f9d00
Sep 27 18:38:56 dist-test-slave-9jll krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534336, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:56.530514 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:56.511558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:34461 (local address 127.2.95.126:39711)
0927 18:38:56.511754 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:56.511759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:56.512914 (+ 1155us) server_negotiation.cc:407] Connection header received
0927 18:38:56.514795 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:56.514801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:56.514885 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:56.514972 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:56.516697 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:56.517421 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:56.519123 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:56.519632 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:56.519785 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:56.523435 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:56.523441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:56.523446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:56.523491 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:56.526736 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:56.527686 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:56.527691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:56.527695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:56.527858 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:56.528378 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:56.528383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:56.528386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:56.528724 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:56.528845 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:56.530038 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0927 18:38:56.530253 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:38:56.531847 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:56.511990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39711 (local address 127.2.95.66:34461)
0927 18:38:56.512735 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:56.512783 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:56.514484 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:56.515235 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:56.515245 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:56.515889 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:56.516507 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:56.516526 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:56.517588 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:56.517592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:56.518956 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:56.518971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:56.519886 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:56.519894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:56.520135 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:56.520824 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:56.520856 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:56.523155 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:56.526957 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:56.526967 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:56.527048 (+ 81us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:56.527538 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:56.528009 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:56.528015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:56.528017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:56.528260 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:56.528888 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:56.528898 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:56.529389 (+ 491us) client_negotiation.cc:769] Sending connection context
0927 18:38:56.529828 (+ 439us) client_negotiation.cc:240] Negotiation successful
0927 18:38:56.530407 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":305,"threads_started":1}
I20260927 18:38:56.533703 8412 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39711
I20260927 18:38:56.534080 8412 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:56.535048 8412 heartbeater.cc:511] Master 127.2.95.126:39711 requested a full tablet report, sending...
I20260927 18:38:56.536787 8089 ts_manager.cc:194] Registered new tserver with Master: 59c3d7f89b8e470a9feb85e0b0f0ea69 (127.2.95.66:36011)
I20260927 18:38:56.537614 8089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:34461
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:38:56.574769 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39711
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:56.909008 8417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:56.909322 8417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:56.909415 8417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:56.917567 8417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:56.917758 8417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:56.917886 8417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:38:56.928293 8417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:56.929627 8417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:56.931058 8417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:56.945053 8425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:56.945086 8423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:56.945008 8422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:56.946753 8417 server_base.cc:1034] running on GCE node
I20260927 18:38:56.947494 8417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:56.948566 8417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:56.949806 8417 hybrid_clock.cc:648] HybridClock initialized: now 1790534336949738 us; error 50 us; skew 500 ppm
I20260927 18:38:56.950181 8417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:56 dist-test-slave-9jll krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534336, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:56.953542 8417 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:38:56.955607 8417 webserver.cc:460] Webserver started at http://127.2.95.67:34065/ using document root <none> and password file <none>
I20260927 18:38:56.956408 8417 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:56.956533 8417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:56.956832 8417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:56.960083 8417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "add99dc108484d9dbe08df37a6cf444a"
format_stamp: "Formatted at 2026-09-27 18:38:56 on dist-test-slave-9jll"
server_key: "c2926008c6320dac868f79cd2b3d1442"
server_key_iv: "29f230b795a75f338c01036721cb7e97"
server_key_version: "encryptionkey@0"
I20260927 18:38:56.960856 8417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "add99dc108484d9dbe08df37a6cf444a"
format_stamp: "Formatted at 2026-09-27 18:38:56 on dist-test-slave-9jll"
server_key: "c2926008c6320dac868f79cd2b3d1442"
server_key_iv: "29f230b795a75f338c01036721cb7e97"
server_key_version: "encryptionkey@0"
I20260927 18:38:56.966472 8417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:38:56.970129 8432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:56.971555 8417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:38:56.971707 8417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "add99dc108484d9dbe08df37a6cf444a"
format_stamp: "Formatted at 2026-09-27 18:38:56 on dist-test-slave-9jll"
server_key: "c2926008c6320dac868f79cd2b3d1442"
server_key_iv: "29f230b795a75f338c01036721cb7e97"
server_key_version: "encryptionkey@0"
I20260927 18:38:56.971907 8417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:56.997133 8417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:57.001714 8417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:57.002043 8417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:57.003118 8417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:57.004570 8417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:57.004653 8417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:57.004769 8417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:57.004822 8417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:57.036034 8278 heartbeater.cc:503] Master 127.2.95.126:39711 was elected leader, sending a full tablet report...
I20260927 18:38:57.037878 8417 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:36583
I20260927 18:38:57.037892 8544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:36583 every 8 connection(s)
I20260927 18:38:57.039304 8417 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:38:57.046703 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 8417
I20260927 18:38:57.046895 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:38:57.047427 2429 external_mini_cluster.cc:1428] Setting key e8b84a22ec182786aca553e701173e68
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534336, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:38:57.060741 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:57.042014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:45453 (local address 127.2.95.126:39711)
0927 18:38:57.042460 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:57.042464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:57.043509 (+ 1045us) server_negotiation.cc:407] Connection header received
0927 18:38:57.045372 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:57.045378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:57.045461 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:57.045553 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:57.047445 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.048326 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.050436 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.050870 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.050972 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:57.054739 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:57.054744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:57.054748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:57.054783 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:57.057224 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.058153 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.058158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.058160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.058307 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.058918 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.058922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.058925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.059231 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:57.059355 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:57.060209 (+ 854us) server_negotiation.cc:299] Negotiation successful
0927 18:38:57.060414 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":238,"threads_started":1}
I20260927 18:38:57.061825 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:57.042522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39711 (local address 127.2.95.67:45453)
0927 18:38:57.043296 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:57.043379 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:57.045100 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:57.045817 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:57.045827 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:57.046376 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:57.047252 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:57.047276 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.048478 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.048483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:57.050283 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:57.050297 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.051102 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.051108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:38:57.051315 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:57.051989 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:38:57.052015 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:38:57.054474 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:38:57.057432 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:57.057443 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:57.057465 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:57.057997 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:57.058482 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:38:57.058486 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:38:57.058489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:38:57.058713 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:38:57.059384 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:38:57.059391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:38:57.059744 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 18:38:57.060057 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 18:38:57.060508 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":312,"threads_started":1}
I20260927 18:38:57.063405 8545 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39711
I20260927 18:38:57.063710 8545 heartbeater.cc:464] Registering TS with master...
I20260927 18:38:57.064409 8545 heartbeater.cc:511] Master 127.2.95.126:39711 requested a full tablet report, sending...
I20260927 18:38:57.065943 8089 ts_manager.cc:194] Registered new tserver with Master: add99dc108484d9dbe08df37a6cf444a (127.2.95.67:36583)
I20260927 18:38:57.066847 8089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:45453
I20260927 18:38:57.075623 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/test-admin.keytab.
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:57.105654 2429 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 18:38:57.111585 8553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:57.108676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.2.95.65:38483)
0927 18:38:57.109354 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:57.109361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:57.109389 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:38:57.109447 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:57.109454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:57.109716 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:57.109882 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:57.111164 (+ 1282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58770: BlockingRecv error: recv got EOF from 127.0.0.1:58770 (error 108)
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":385,"threads_started":1}
W20260927 18:38:57.111809 2429 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:57.113459 2429 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:57.114933 2429 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:57.131767 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:57.116644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.2.95.65:38483)
0927 18:38:57.116813 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:57.116819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:57.117132 (+ 313us) server_negotiation.cc:407] Connection header received
0927 18:38:57.117377 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:57.117382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:57.117446 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:57.117531 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:57.118609 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.119705 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.121052 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.121621 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.121784 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:57.125774 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:57.125788 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:57.125816 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:57.125857 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:57.128674 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.129576 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.129584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.129586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.129764 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.130212 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.130216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.130219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.130642 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:57.130930 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:57.131264 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 18:38:57.131505 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 18:38:57.138327 8553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:57.136319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.2.95.65:38483)
0927 18:38:57.136533 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:57.136540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:57.136638 (+ 98us) server_negotiation.cc:407] Connection header received
0927 18:38:57.136823 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:57.136827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:57.137009 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:57.137107 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:57.138076 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58776: BlockingRecv error: recv got EOF from 127.0.0.1:58776 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 18:38:57.138460 2429 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:57.140074 2429 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:57.141311 2429 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:57.156631 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:57.142768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.2.95.65:38483)
0927 18:38:57.143081 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:57.143087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:57.143245 (+ 158us) server_negotiation.cc:407] Connection header received
0927 18:38:57.143363 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:57.143373 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:57.143442 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:57.143528 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:57.144585 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.145389 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.146733 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.147236 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.147367 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:57.151268 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:57.151272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:57.151276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:57.151316 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:57.154108 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.154798 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.154802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.154804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.154935 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.155273 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.155277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.155279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.155591 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:57.155819 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:57.156119 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 18:38:57.156329 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 18:38:57.161327 8553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:38:57.159411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.2.95.65:38483)
0927 18:38:57.159620 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:57.159626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:57.159825 (+ 199us) server_negotiation.cc:407] Connection header received
0927 18:38:57.160109 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:57.160114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:57.160189 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:57.160278 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:57.161092 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58798: BlockingRecv error: recv got EOF from 127.0.0.1:58798 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 18:38:57.161583 2429 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790534284715420-2429-0/krb5kdc/krb5cc' of type FILE
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:57.163189 2429 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:57.164459 2429 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:38:57.179708 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:57.165712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.2.95.65:38483)
0927 18:38:57.165894 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:57.165900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:57.166079 (+ 179us) server_negotiation.cc:407] Connection header received
0927 18:38:57.166283 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:57.166287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:57.166356 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:57.166445 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:57.167298 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.168029 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.169406 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.169992 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.170095 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:57.173839 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:57.173845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:57.173848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:57.173887 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:57.176498 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.177372 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.177377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.177379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.177519 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.178055 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.178059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.178062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.178392 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:57.178660 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:57.179082 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 18:38:57.179368 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 18:38:57.181298 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 8150
I20260927 18:38:57.194487 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:57.182032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.2.95.65:38483)
0927 18:38:57.182186 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:57.182190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:57.182627 (+ 437us) server_negotiation.cc:407] Connection header received
0927 18:38:57.182740 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:57.182744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:57.182807 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:57.182977 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:57.184099 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.184812 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.186220 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.186682 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.186792 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:57.189063 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:57.189068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:57.189072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:57.189110 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:57.191753 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.192468 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.192472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.192475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.192618 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.192992 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.192996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.192998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.193288 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:57.193509 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:57.194036 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 18:38:57.194223 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:38:57.195053 8273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:57.278728 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 8150
I20260927 18:38:57.291934 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 8284
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:38:57.308279 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:57.292825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.2.95.66:36011)
0927 18:38:57.293303 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:57.293310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:57.293361 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:38:57.293427 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:57.293433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:57.293735 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:57.293891 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:57.294974 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.296011 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.297383 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.298112 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.298227 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:57.302140 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:57.302154 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:57.302171 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:57.302210 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:57.305140 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.306011 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.306019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.306022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.306155 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.306529 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.306532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.306535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.307008 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:57.307338 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:57.307765 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 18:38:57.307982 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":222,"threads_started":1}
I20260927 18:38:57.309615 8407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:57.396250 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 8284
I20260927 18:38:57.408918 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 8417
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:38:57.425627 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:57.409840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.2.95.67:36583)
0927 18:38:57.410456 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:57.410463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:57.410496 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:38:57.410556 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:57.410564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:57.410774 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:57.410917 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:57.412166 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.413548 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.414932 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:57.415510 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:57.415622 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:57.419743 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:57.419760 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:57.419778 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:57.419824 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:57.422851 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.423651 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.423659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.423662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.423789 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:57.424173 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:57.424177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:57.424179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:57.424545 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:57.424774 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:57.425056 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 18:38:57.425320 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":301,"threads_started":1}
I20260927 18:38:57.426899 8540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:57.510440 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 8417
I20260927 18:38:57.522643 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 8057
I20260927 18:38:57.523798 8119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:38:57.624878 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 8057
2026-09-27T18:38:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3917 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8573](info): set up 2 sockets
Sep 27 18:38:57 dist-test-slave-9jll krb5kdc[8573](info): commencing operation
krb5kdc: starting...
W20260927 18:38:58.634349 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.005s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:38:58 dist-test-slave-9jll krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:38:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:38:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.126.keytab.
Entry for principal oryx/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.126.keytab.
I20260927 18:38:58.849282 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:39071
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.126.keytab
--principal=oryx/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:39071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:59.176357 8586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:59.176672 8586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:59.176785 8586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:59.184162 8586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:38:59.184309 8586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:59.184422 8586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:59.184516 8586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:38:59.184573 8586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:38:59.194622 8586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:39071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.126.keytab
--principal=oryx/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:39071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:59.195981 8586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:59.197260 8586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:59.208880 8594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:59.209324 8591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:59.209362 8592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:59.210175 8586 server_base.cc:1034] running on GCE node
I20260927 18:38:59.210994 8586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:59.212483 8586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:59.213727 8586 hybrid_clock.cc:648] HybridClock initialized: now 1790534339213650 us; error 37 us; skew 500 ppm
I20260927 18:38:59.214073 8586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:59 dist-test-slave-9jll krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534339, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:59.226771 8586 init.cc:377] Logged in from keytab as oryx/127.2.95.126@KRBTEST.COM (short username oryx)
I20260927 18:38:59.228432 8586 webserver.cc:460] Webserver started at http://127.2.95.126:43429/ using document root <none> and password file <none>
I20260927 18:38:59.229168 8586 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:59.229281 8586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:59.229542 8586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:59.232637 8586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "58a7477d2f2741d48e6705163e08facd"
format_stamp: "Formatted at 2026-09-27 18:38:59 on dist-test-slave-9jll"
server_key: "1850ad832137e301d671c7c77bc4c4be"
server_key_iv: "ed3ec30e61f5ebd48945e9749e6ec165"
server_key_version: "encryptionkey@0"
I20260927 18:38:59.233382 8586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "58a7477d2f2741d48e6705163e08facd"
format_stamp: "Formatted at 2026-09-27 18:38:59 on dist-test-slave-9jll"
server_key: "1850ad832137e301d671c7c77bc4c4be"
server_key_iv: "ed3ec30e61f5ebd48945e9749e6ec165"
server_key_version: "encryptionkey@0"
I20260927 18:38:59.238780 8586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 18:38:59.242559 8601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:59.244169 8586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:38:59.244350 8586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "58a7477d2f2741d48e6705163e08facd"
format_stamp: "Formatted at 2026-09-27 18:38:59 on dist-test-slave-9jll"
server_key: "1850ad832137e301d671c7c77bc4c4be"
server_key_iv: "ed3ec30e61f5ebd48945e9749e6ec165"
server_key_version: "encryptionkey@0"
I20260927 18:38:59.244567 8586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:59.290170 8586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:59.294979 8586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:59.295271 8586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:59.313258 8586 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:39071
I20260927 18:38:59.313282 8652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:39071 every 8 connection(s)
I20260927 18:38:59.314658 8586 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:38:59.318749 8653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:38:59.320849 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 8586
I20260927 18:38:59.321061 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:38:59.321708 2429 external_mini_cluster.cc:1428] Setting key 327a87a90b1dc92bfc5beded51eeee94
I20260927 18:38:59.330791 8653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd: Bootstrap starting.
I20260927 18:38:59.334812 8653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd: Neither blocks nor log segments found. Creating new log.
I20260927 18:38:59.336202 8653 log.cc:824] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:38:59 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:38:59.339975 8653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd: No bootstrap required, opened a new log
I20260927 18:38:59.346525 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:59.324024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.2.95.126:39071)
0927 18:38:59.324867 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:59.324881 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:59.324926 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:38:59.326638 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:59.326661 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:59.327105 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:59.327479 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:59.329271 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:59.330723 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:59.332561 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:38:59.333325 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:38:59.333482 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:38:59.338487 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:38:59.338503 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:38:59.338522 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:38:59.338564 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:38:59.341769 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:59.342701 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:59.342708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:59.342716 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:59.343003 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:38:59.343420 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:38:59.343424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:38:59.343426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:38:59.344105 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:38:59.344350 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 18:38:59.344823 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 18:38:59.345331 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":377,"threads_started":1}
I20260927 18:38:59.349344 8653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } }
I20260927 18:38:59.349843 8653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:38:59.349956 8653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a7477d2f2741d48e6705163e08facd, State: Initialized, Role: FOLLOWER
I20260927 18:38:59.350512 8653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } }
I20260927 18:38:59.350713 8653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:38:59.350843 8653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:38:59.350981 8653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:38:59.353247 8653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } }
I20260927 18:38:59.353865 8653 leader_election.cc:302] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58a7477d2f2741d48e6705163e08facd; no voters:
I20260927 18:38:59.354453 8653 leader_election.cc:288] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:38:59.354578 8658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:38:59.355237 8658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 1 LEADER]: Becoming Leader. State: Replica: 58a7477d2f2741d48e6705163e08facd, State: Running, Role: LEADER
I20260927 18:38:59.355803 8658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } }
I20260927 18:38:59.356591 8653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:38:59.358680 8660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a7477d2f2741d48e6705163e08facd. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } }
I20260927 18:38:59.358651 8659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } }
I20260927 18:38:59.359119 8660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:59.359195 8659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: This master's current role is: LEADER
I20260927 18:38:59.361243 8663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:38:59.366503 8663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:38:59.376602 8663 catalog_manager.cc:1347] Generated new cluster ID: dac25e4c532440a4ba8372b8b7af6383
I20260927 18:38:59.376724 8663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:38:59.395573 8663 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:38:59.396921 8663 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:38:59.411017 8663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd: Generated new TSK 0
I20260927 18:38:59.412042 8663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab.
Entry for principal oryx/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab.
I20260927 18:38:59.481652 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39071
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab
--principal=oryx/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:38:59.849203 8679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:38:59.849505 8679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:38:59.849602 8679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:38:59.857213 8679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:38:59.857340 8679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:38:59.857482 8679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:38:59.868024 8679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab
--principal=oryx/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:38:59.869426 8679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:38:59.870970 8679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:38:59.886461 8687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:59.886533 8684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:38:59.886828 8685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:38:59.888034 8679 server_base.cc:1034] running on GCE node
I20260927 18:38:59.888826 8679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:38:59.890024 8679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:38:59.891278 8679 hybrid_clock.cc:648] HybridClock initialized: now 1790534339891207 us; error 63 us; skew 500 ppm
I20260927 18:38:59.891673 8679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:38:59 dist-test-slave-9jll krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534339, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:38:59.895464 8679 init.cc:377] Logged in from keytab as oryx/127.2.95.65@KRBTEST.COM (short username oryx)
I20260927 18:38:59.897330 8679 webserver.cc:460] Webserver started at http://127.2.95.65:34901/ using document root <none> and password file <none>
I20260927 18:38:59.898144 8679 fs_manager.cc:359] Metadata directory not provided
I20260927 18:38:59.898272 8679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:38:59.898567 8679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:38:59.901712 8679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "52c5acedd2a3495c83ed550b02b06660"
format_stamp: "Formatted at 2026-09-27 18:38:59 on dist-test-slave-9jll"
server_key: "279f84570e30cb1d770ed8867298d203"
server_key_iv: "c3fe08bd56a511a6b48640fa2f612655"
server_key_version: "encryptionkey@0"
I20260927 18:38:59.902556 8679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "52c5acedd2a3495c83ed550b02b06660"
format_stamp: "Formatted at 2026-09-27 18:38:59 on dist-test-slave-9jll"
server_key: "279f84570e30cb1d770ed8867298d203"
server_key_iv: "c3fe08bd56a511a6b48640fa2f612655"
server_key_version: "encryptionkey@0"
I20260927 18:38:59.908283 8679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:38:59.912707 8694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:59.914609 8679 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:38:59.914808 8679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "52c5acedd2a3495c83ed550b02b06660"
format_stamp: "Formatted at 2026-09-27 18:38:59 on dist-test-slave-9jll"
server_key: "279f84570e30cb1d770ed8867298d203"
server_key_iv: "c3fe08bd56a511a6b48640fa2f612655"
server_key_version: "encryptionkey@0"
I20260927 18:38:59.915005 8679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:38:59.946904 8679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:38:59.952181 8679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:38:59.952446 8679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:38:59.953449 8679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:38:59.954937 8679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:38:59.955024 8679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:59.955116 8679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:38:59.955158 8679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:38:59.989317 8679 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:39523
I20260927 18:38:59.989393 8806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:39523 every 8 connection(s)
I20260927 18:38:59.990675 8679 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:38:59.996762 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 8679
I20260927 18:38:59.996999 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:38:59.997848 2429 external_mini_cluster.cc:1428] Setting key 0db5ae7d241ae1375d24f2ac58b2f829
Sep 27 18:39:00 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534339, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:00.017841 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:59.993252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:42829 (local address 127.2.95.126:39071)
0927 18:38:59.993723 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 18:38:59.993729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:38:59.994916 (+ 1187us) server_negotiation.cc:407] Connection header received
0927 18:38:59.997069 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:38:59.997076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:38:59.997180 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:38:59.997285 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:38:59.999360 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:00.000249 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:00.003145 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:00.003569 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:00.003694 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:00.009360 (+ 5666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:00.009369 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:00.009375 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:00.009423 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:00.013045 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:00.014892 (+ 1847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:00.014899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:00.014902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:00.015112 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:00.015767 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:00.015773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:00.015776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:00.016157 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:00.016310 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:00.017142 (+ 832us) server_negotiation.cc:299] Negotiation successful
0927 18:39:00.017412 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":196,"threads_started":1}
I20260927 18:39:00.018805 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:38:59.993964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.65:42829)
0927 18:38:59.994724 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 18:38:59.994766 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:38:59.996631 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:38:59.997621 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:38:59.997637 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:38:59.998454 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:38:59.999147 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:38:59.999174 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:00.000417 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:00.000422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:00.002868 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:00.002895 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:00.003848 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:00.003867 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:00.004177 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:00.005421 (+ 1244us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:00.005456 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:00.008991 (+ 3535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:00.013413 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:00.013428 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:00.013451 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:00.014638 (+ 1187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:00.015304 (+ 666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:00.015309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:00.015312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:00.015579 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:00.016305 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:00.016314 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:00.016706 (+ 392us) client_negotiation.cc:769] Sending connection context
0927 18:39:00.016990 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 18:39:00.017560 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":292,"threads_started":1}
I20260927 18:39:00.020620 8807 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39071
I20260927 18:39:00.020996 8807 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.2.95.66@KRBTEST.COM; defaulting to no policy
I20260927 18:39:00.021922 8807 heartbeater.cc:511] Master 127.2.95.126:39071 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.66@KRBTEST.COM" created.
I20260927 18:39:00.024505 8618 ts_manager.cc:194] Registered new tserver with Master: 52c5acedd2a3495c83ed550b02b06660 (127.2.95.65:39523)
I20260927 18:39:00.026573 8618 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:42829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab.
Entry for principal oryx/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab.
I20260927 18:39:00.050879 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39071
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab
--principal=oryx/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:00.382550 8813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:00.382871 8813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:00.383006 8813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:00.390671 8813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:00.390797 8813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:00.390944 8813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:39:00.401001 8813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab
--principal=oryx/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:00.402424 8813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:00.403736 8813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:00.417659 8821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:00.417760 8818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:00.417717 8819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:00.418807 8813 server_base.cc:1034] running on GCE node
I20260927 18:39:00.419454 8813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:00.420770 8813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:00.422010 8813 hybrid_clock.cc:648] HybridClock initialized: now 1790534340421935 us; error 67 us; skew 500 ppm
I20260927 18:39:00.422345 8813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:00 dist-test-slave-9jll krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534340, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:00.425563 8813 init.cc:377] Logged in from keytab as oryx/127.2.95.66@KRBTEST.COM (short username oryx)
I20260927 18:39:00.427050 8813 webserver.cc:460] Webserver started at http://127.2.95.66:35063/ using document root <none> and password file <none>
I20260927 18:39:00.427732 8813 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:00.427845 8813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:00.428201 8813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:00.431295 8813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36"
format_stamp: "Formatted at 2026-09-27 18:39:00 on dist-test-slave-9jll"
server_key: "0fa6d7cda388e3ed663fcd38d5a24941"
server_key_iv: "1472fc326598cf20c385e8e196003297"
server_key_version: "encryptionkey@0"
I20260927 18:39:00.432087 8813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36"
format_stamp: "Formatted at 2026-09-27 18:39:00 on dist-test-slave-9jll"
server_key: "0fa6d7cda388e3ed663fcd38d5a24941"
server_key_iv: "1472fc326598cf20c385e8e196003297"
server_key_version: "encryptionkey@0"
I20260927 18:39:00.437734 8813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 18:39:00.441382 8828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:00.442941 8813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 18:39:00.443095 8813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36"
format_stamp: "Formatted at 2026-09-27 18:39:00 on dist-test-slave-9jll"
server_key: "0fa6d7cda388e3ed663fcd38d5a24941"
server_key_iv: "1472fc326598cf20c385e8e196003297"
server_key_version: "encryptionkey@0"
I20260927 18:39:00.443274 8813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:00.465716 8813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:00.469969 8813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:00.470239 8813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:00.471206 8813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:00.472563 8813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:00.472640 8813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:00.472723 8813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:00.472764 8813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:00.504393 8813 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:36525
I20260927 18:39:00.504443 8940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:36525 every 8 connection(s)
I20260927 18:39:00.505622 8813 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:39:00.512476 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 8813
I20260927 18:39:00.512666 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:39:00.513234 2429 external_mini_cluster.cc:1428] Setting key 258cfde789a2c9c74c15e712ff88636b
Sep 27 18:39:00 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534340, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:00.527207 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:00.508386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:53817 (local address 127.2.95.126:39071)
0927 18:39:00.508584 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:00.508589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:00.509643 (+ 1054us) server_negotiation.cc:407] Connection header received
0927 18:39:00.511522 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:00.511527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:00.511599 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:00.511693 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:00.513250 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:00.514231 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:00.516040 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:00.516704 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:00.516839 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:00.520729 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:00.520736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:00.520741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:00.520781 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:00.523840 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:00.524629 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:00.524634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:00.524637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:00.524801 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:00.525318 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:00.525323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:00.525325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:00.525704 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:00.525856 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:00.526619 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 18:39:00.526885 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:39:00.528146 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:00.508783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.66:53817)
0927 18:39:00.509486 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:00.509525 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:00.511235 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:00.511889 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:00.511897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:00.512438 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:00.513068 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:00.513086 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:00.514381 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:00.514386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:00.515893 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:00.515906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:00.516972 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:00.516979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:00.517215 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:00.517910 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:00.517938 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:00.520466 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:00.524031 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:00.524039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:00.524054 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:00.524500 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:00.524990 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:00.524994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:00.524996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:00.525212 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:00.525836 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:00.525842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:00.526270 (+ 428us) client_negotiation.cc:769] Sending connection context
0927 18:39:00.526522 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 18:39:00.526949 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":273,"threads_started":1}
I20260927 18:39:00.529870 8941 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39071
I20260927 18:39:00.530241 8941 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:00.530966 8941 heartbeater.cc:511] Master 127.2.95.126:39071 requested a full tablet report, sending...
I20260927 18:39:00.532510 8618 ts_manager.cc:194] Registered new tserver with Master: ba8c6a10b9e94dc0bea3e8e25c261e36 (127.2.95.66:36525)
WARNING: no policy specified for oryx/127.2.95.67@KRBTEST.COM; defaulting to no policy
I20260927 18:39:00.533425 8618 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:53817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab.
Entry for principal oryx/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab.
I20260927 18:39:00.561743 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39071
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab
--principal=oryx/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:00.888001 8946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:00.888314 8946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:00.888458 8946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:00.895871 8946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:00.896001 8946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:00.896116 8946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:39:00.906160 8946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab
--principal=oryx/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:00.907434 8946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:00.908700 8946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:00.922748 8952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:00.922777 8951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:00.922819 8954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:00.923452 8946 server_base.cc:1034] running on GCE node
I20260927 18:39:00.924077 8946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:00.925076 8946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:00.926393 8946 hybrid_clock.cc:648] HybridClock initialized: now 1790534340926284 us; error 49 us; skew 500 ppm
I20260927 18:39:00.926842 8946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:00 dist-test-slave-9jll krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534340, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:00.930418 8946 init.cc:377] Logged in from keytab as oryx/127.2.95.67@KRBTEST.COM (short username oryx)
I20260927 18:39:00.931962 8946 webserver.cc:460] Webserver started at http://127.2.95.67:38051/ using document root <none> and password file <none>
I20260927 18:39:00.932691 8946 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:00.932799 8946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:00.933095 8946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:00.936079 8946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "db6625f555d64f52b480f453712da79a"
format_stamp: "Formatted at 2026-09-27 18:39:00 on dist-test-slave-9jll"
server_key: "7e889087967ffc9319b5e9c13100acc7"
server_key_iv: "cd315c8a6aa6056d49ee38fea029aea0"
server_key_version: "encryptionkey@0"
I20260927 18:39:00.936820 8946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "db6625f555d64f52b480f453712da79a"
format_stamp: "Formatted at 2026-09-27 18:39:00 on dist-test-slave-9jll"
server_key: "7e889087967ffc9319b5e9c13100acc7"
server_key_iv: "cd315c8a6aa6056d49ee38fea029aea0"
server_key_version: "encryptionkey@0"
I20260927 18:39:00.942328 8946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 18:39:00.946522 8961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:00.948267 8946 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:39:00.948418 8946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "db6625f555d64f52b480f453712da79a"
format_stamp: "Formatted at 2026-09-27 18:39:00 on dist-test-slave-9jll"
server_key: "7e889087967ffc9319b5e9c13100acc7"
server_key_iv: "cd315c8a6aa6056d49ee38fea029aea0"
server_key_version: "encryptionkey@0"
I20260927 18:39:00.948619 8946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:00.979451 8946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:00.983760 8946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:00.984041 8946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:00.984998 8946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:00.986457 8946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:00.986534 8946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:00.986624 8946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:00.986673 8946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:01.017293 8946 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:37053
I20260927 18:39:01.017323 9073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:37053 every 8 connection(s)
I20260927 18:39:01.018661 8946 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:39:01.022238 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 8946
I20260927 18:39:01.022421 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:39:01.022986 2429 external_mini_cluster.cc:1428] Setting key 54a2baadbc55d6b9339fc3eb1b2a86ed
I20260927 18:39:01.029964 8807 heartbeater.cc:503] Master 127.2.95.126:39071 was elected leader, sending a full tablet report...
Sep 27 18:39:01 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534340, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:01.039187 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.021203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:53315 (local address 127.2.95.126:39071)
0927 18:39:01.021417 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:01.021423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:01.022807 (+ 1384us) server_negotiation.cc:407] Connection header received
0927 18:39:01.024684 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:01.024689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:01.024774 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:01.024857 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:01.026487 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.027210 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.028836 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.029262 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.029374 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.033382 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:01.033388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:01.033428 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:01.033469 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:01.036129 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.036896 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.036900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.036902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.037036 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.037456 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.037460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.037462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.037786 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:01.037912 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:01.038619 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 18:39:01.038828 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:39:01.040207 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.021797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.67:53315)
0927 18:39:01.022640 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:01.022686 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:01.024410 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:01.025122 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:01.025132 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:01.025695 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:01.026313 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.026330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.027365 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.027369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.028680 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.028693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.029499 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.029507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.029792 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.030570 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:01.030603 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:01.033149 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:01.036299 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.036307 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.036325 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.036762 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.037149 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.037153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.037156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.037340 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.037911 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:01.037919 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:01.038275 (+ 356us) client_negotiation.cc:769] Sending connection context
0927 18:39:01.038509 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 18:39:01.039066 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":356,"threads_started":1}
I20260927 18:39:01.041625 9074 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39071
I20260927 18:39:01.041937 9074 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:01.042630 9074 heartbeater.cc:511] Master 127.2.95.126:39071 requested a full tablet report, sending...
I20260927 18:39:01.044054 8618 ts_manager.cc:194] Registered new tserver with Master: db6625f555d64f52b480f453712da79a (127.2.95.67:37053)
I20260927 18:39:01.044883 8618 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:53315
I20260927 18:39:01.050627 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.95.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.125.keytab.
Entry for principal oryx/127.2.95.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.125.keytab.
I20260927 18:39:01.101907 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.125:43979
--webserver_interface=127.2.95.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.125.keytab
--principal=oryx/127.2.95.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:39071,127.2.95.125:43979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:01.435453 9079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:01.435751 9079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:01.435837 9079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:01.443403 9079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:39:01.443526 9079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:01.443569 9079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:01.443624 9079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:39:01.443667 9079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:39:01.454073 9079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:39071,127.2.95.125:43979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.125.keytab
--principal=oryx/127.2.95.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.125:43979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.95.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
W20260927 18:39:01.454411 9079 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.95.126:39071,127.2.95.125:43979'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 18:39:01.455436 9079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:01.456775 9079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:01.468773 9085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:01.468801 9084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:01.468833 9087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:01.469717 9079 server_base.cc:1034] running on GCE node
I20260927 18:39:01.470546 9079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:01.472059 9079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:01.473265 9079 hybrid_clock.cc:648] HybridClock initialized: now 1790534341473209 us; error 40 us; skew 500 ppm
I20260927 18:39:01.473624 9079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:01 dist-test-slave-9jll krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534341, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:01.476938 9079 init.cc:377] Logged in from keytab as oryx/127.2.95.125@KRBTEST.COM (short username oryx)
I20260927 18:39:01.478572 9079 webserver.cc:460] Webserver started at http://127.2.95.125:37687/ using document root <none> and password file <none>
I20260927 18:39:01.479271 9079 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:01.479389 9079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:01.479701 9079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:01.482775 9079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data/instance:
uuid: "9c9618bf0f8648ed987531e4f6839082"
format_stamp: "Formatted at 2026-09-27 18:39:01 on dist-test-slave-9jll"
server_key: "446c62010ac963436df4418276b8efaa"
server_key_iv: "fe9390d9755cce893d021a49f408dc79"
server_key_version: "encryptionkey@0"
I20260927 18:39:01.483578 9079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/wal/instance:
uuid: "9c9618bf0f8648ed987531e4f6839082"
format_stamp: "Formatted at 2026-09-27 18:39:01 on dist-test-slave-9jll"
server_key: "446c62010ac963436df4418276b8efaa"
server_key_iv: "fe9390d9755cce893d021a49f408dc79"
server_key_version: "encryptionkey@0"
I20260927 18:39:01.489014 9079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:39:01.492731 9094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:01.494299 9079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 18:39:01.494478 9079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/wal
uuid: "9c9618bf0f8648ed987531e4f6839082"
format_stamp: "Formatted at 2026-09-27 18:39:01 on dist-test-slave-9jll"
server_key: "446c62010ac963436df4418276b8efaa"
server_key_iv: "fe9390d9755cce893d021a49f408dc79"
server_key_version: "encryptionkey@0"
I20260927 18:39:01.494693 9079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:01.526252 9079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:01.530838 9079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:01.531106 9079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:01.536248 8941 heartbeater.cc:503] Master 127.2.95.126:39071 was elected leader, sending a full tablet report...
I20260927 18:39:01.549764 9079 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.125:43979
I20260927 18:39:01.549782 9145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.125:43979 every 8 connection(s)
I20260927 18:39:01.551169 9079 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
I20260927 18:39:01.552743 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 9079
I20260927 18:39:01.555986 9146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:01.564466 9146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:39:01 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.95.125@KRBTEST.COM
Sep 27 18:39:01 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534341, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.125@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:01.578608 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.555139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.2.95.125:43979)
0927 18:39:01.555894 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:01.555906 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:01.555947 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:39:01.557597 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:01.557624 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:01.558051 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:01.558372 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:01.560162 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.561619 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.563591 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.564316 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.564492 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.569550 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:01.569561 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:01.569580 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:01.569627 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:01.573226 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.573990 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.573997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.574004 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.574161 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.575082 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.575087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.575091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.575753 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:01.576003 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:01.576606 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 18:39:01.577067 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":335,"threads_started":1}
I20260927 18:39:01.584347 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.567699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.2.95.126:39071)
0927 18:39:01.568158 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:01.568163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:01.569268 (+ 1105us) server_negotiation.cc:407] Connection header received
0927 18:39:01.569473 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:01.569478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:01.569566 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:01.569700 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:01.571219 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.572390 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.573923 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.574440 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.574570 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.578553 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:01.578559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:01.578563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:01.578600 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:01.581646 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.582385 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.582389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.582391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.582539 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.582978 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.582982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.582984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.583314 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:01.583434 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:01.583874 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 18:39:01.584084 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":248,"threads_started":1}
I20260927 18:39:01.584372 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.567680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:46570)
0927 18:39:01.569094 (+ 1414us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:01.569150 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:01.569334 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:01.569806 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:01.569823 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:01.570670 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:01.571032 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.571042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.572560 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.572564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.573772 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.573791 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.574633 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.574637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.574784 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.575450 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:01.575468 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:01.578321 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:01.581827 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.581835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.581840 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.582261 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.582628 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.582632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.582635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.582771 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.583441 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:01.583448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:01.583613 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 18:39:01.583763 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 18:39:01.584090 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"thread_start_us":1016,"threads_started":1}
I20260927 18:39:01.586192 9146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:39:01 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534341, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.125@KRBTEST.COM for oryx/127.2.95.125@KRBTEST.COM
I20260927 18:39:01.598989 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.586826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:43979 (local address 127.0.0.1:36154)
0927 18:39:01.587029 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:01.587049 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:01.587154 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:01.587405 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:01.587418 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:01.587771 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:01.587901 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.587909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.588847 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.588851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.589892 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.589905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.590583 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.590589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.590744 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.591323 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:01.591400 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:01.593543 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:01.596405 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.596409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.596412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.596764 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.597271 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.597275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.597278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.597451 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.598092 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:01.598104 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:01.598259 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 18:39:01.598399 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 18:39:01.598643 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 18:39:01.599334 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.586937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.2.95.125:43979)
0927 18:39:01.587082 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:01.587087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:01.587113 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:39:01.587249 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:01.587253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:01.587306 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:01.587379 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:01.588029 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.588720 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.590035 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.590435 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.590548 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.593768 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:01.593773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:01.593776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:01.593805 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:01.596253 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.596909 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.596913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.596916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.597107 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.597560 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.597564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.597566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.597890 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:01.598086 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:01.598517 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 18:39:01.598846 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":61}
I20260927 18:39:01.603976 9146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: Bootstrap starting.
I20260927 18:39:01.607191 9146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:01.608263 9146 log.cc:824] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:01.611258 9146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: No bootstrap required, opened a new log
I20260927 18:39:01.618254 9146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } }
I20260927 18:39:01.618609 9146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:01.618706 9146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c9618bf0f8648ed987531e4f6839082, State: Initialized, Role: FOLLOWER
I20260927 18:39:01.619364 9146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } }
I20260927 18:39:01.620460 9153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } } }
I20260927 18:39:01.620772 9153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:39:01.621420 9146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:39:01.630477 9164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:39:01.630633 9164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:39:01.633617 9079 master_runner.cc:428] Detected that this master 9c9618bf0f8648ed987531e4f6839082 is joining an existing cluster
I20260927 18:39:01.633764 9079 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.95.125:43979
I20260927 18:39:01.645862 9153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:39:01.646222 9153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } }
I20260927 18:39:01.647781 9153 leader_election.cc:288] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58a7477d2f2741d48e6705163e08facd (127.2.95.126:39071)
I20260927 18:39:01.649148 8628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c9618bf0f8648ed987531e4f6839082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a7477d2f2741d48e6705163e08facd" is_pre_election: true
I20260927 18:39:01.650769 9096 leader_election.cc:302] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 9c9618bf0f8648ed987531e4f6839082; no voters: 58a7477d2f2741d48e6705163e08facd
I20260927 18:39:01.651942 9153 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 18:39:01.656515 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.639479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.2.95.125:43979)
0927 18:39:01.639709 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:01.639715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:01.642357 (+ 2642us) server_negotiation.cc:407] Connection header received
0927 18:39:01.642553 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:01.642558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:01.642620 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:01.642682 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:01.643968 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.644681 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.646535 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.647200 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.647288 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.649851 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:01.649858 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:01.649862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:01.649897 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:01.653412 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.654330 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.654335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.654339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.654474 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.654876 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.654880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.654884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.655262 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:01.655420 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:01.655917 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 18:39:01.656137 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:39:01.657519 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.639184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:43979 (local address 127.0.0.1:36160)
0927 18:39:01.642258 (+ 3074us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:01.642277 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:01.642446 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:01.642846 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:01.642850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:01.643519 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:01.643848 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.643862 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.644857 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.644862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.646380 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.646396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.647593 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.647599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.647788 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.648426 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:01.648448 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:01.649622 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:01.653643 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.653705 (+ 62us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.653713 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.654227 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.654610 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.654614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.654618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.654763 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.655495 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:01.655510 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:01.655660 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 18:39:01.656697 (+ 1037us) client_negotiation.cc:240] Negotiation successful
0927 18:39:01.657061 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2924,"mutex_wait_us":22,"thread_start_us":2718,"threads_started":1}
I20260927 18:39:01.660569 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.638495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.2.95.126:39071)
0927 18:39:01.638681 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:01.638687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:01.638873 (+ 186us) server_negotiation.cc:407] Connection header received
0927 18:39:01.639133 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:01.639139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:01.639217 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:01.639278 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:01.640716 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.641786 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.643864 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.644542 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.644680 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.654208 (+ 9528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:01.654216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:01.654221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:01.654264 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:01.657571 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.658310 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.658314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.658317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.658486 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.658947 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.658952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.658956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.659314 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:01.659442 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:01.660034 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 18:39:01.660246 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:39:01.660740 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.638104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:46584)
0927 18:39:01.638761 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:01.638781 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:01.638985 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:01.639484 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:01.639489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:01.640154 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:01.640531 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.640546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.641995 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.642000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.643667 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.643680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.644741 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.644747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.645021 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.652644 (+ 7623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:01.652673 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:01.653951 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:01.657753 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.657758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.657762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.658182 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.658642 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.658646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.658649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.658808 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.659550 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:01.659555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:01.659764 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 18:39:01.659897 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 18:39:01.660207 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":45,"thread_start_us":580,"threads_started":1}
WARNING: no policy specified for oryx/127.2.95.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.124@KRBTEST.COM" created.
Sep 27 18:39:01 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534339, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.126@KRBTEST.COM for oryx/127.2.95.125@KRBTEST.COM
I20260927 18:39:01.687003 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.672312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.2.95.125:43979)
0927 18:39:01.672503 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:01.672510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:01.673273 (+ 763us) server_negotiation.cc:407] Connection header received
0927 18:39:01.673635 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:01.673639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:01.673747 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:01.673829 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:01.674978 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.675800 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.677132 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.677752 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.677886 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.681214 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:01.681220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:01.681223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:01.681255 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:01.684135 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.684855 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.684859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.684862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.684998 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.685384 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.685387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.685390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.685663 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:01.685840 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:01.686374 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 18:39:01.686609 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:39:01.687143 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.672303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:43979 (local address 127.0.0.1:36162)
0927 18:39:01.673124 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:01.673162 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:01.673416 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:01.673907 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:01.673924 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:01.674579 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:01.674803 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.674812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.675937 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.675942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.676955 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.676968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.677930 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.677937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.678135 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.678803 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:01.678820 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:01.681013 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:01.684285 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.684302 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.684311 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.684738 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.685114 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.685119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.685123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.685277 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.685840 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:01.685849 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:01.686091 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 18:39:01.686280 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 18:39:01.686693 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":357,"threads_started":1}
I20260927 18:39:01.689320 8618 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.95.125:43979: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9c9618bf0f8648ed987531e4f6839082"
member_type: NON_VOTER
last_known_addr {
host: "127.2.95.125"
port: 43979
}
attrs {
promote: true
}
}
dest_uuid: "58a7477d2f2741d48e6705163e08facd"
cas_config_opid_index: -1
I20260927 18:39:01.690555 8618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } }
I20260927 18:39:01.693887 9177 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9c9618bf0f8648ed987531e4f6839082 (127.2.95.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } }
I20260927 18:39:01.694947 9121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.124.keytab.
Entry for principal oryx/127.2.95.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.124.keytab.
I20260927 18:39:01.697892 9121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 FOLLOWER]: Refusing update from remote peer 58a7477d2f2741d48e6705163e08facd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 18:39:01.698215 9177 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.95.125:43979
I20260927 18:39:01.698837 9176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } }
I20260927 18:39:01.699189 9079 master.cc:537] Master@127.2.95.125:43979 shutting down...
I20260927 18:39:01.699312 9176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:01.699616 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.124:39795
--webserver_interface=127.2.95.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.124.keytab
--principal=oryx/127.2.95.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:39071,127.2.95.125:43979,127.2.95.124:39795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:39:01.706213 9177 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:39:01.709764 8603 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 18:39:01.714337 9079 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 18:39:01.715421 8603 consensus_peers.cc:487] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd -> Peer 9c9618bf0f8648ed987531e4f6839082 (127.2.95.125:43979): Couldn't send request to peer 9c9618bf0f8648ed987531e4f6839082. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 18:39:01.716070 9079 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:01.716336 9079 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: stopping tablet replica
I20260927 18:39:01.724036 9079 master.cc:552] Master@127.2.95.125:43979 shutdown complete.
I20260927 18:39:01.724251 9079 master_runner.cc:305] Clearing existing system tablet
I20260927 18:39:01.724812 9079 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:01.724985 9079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:01.728199 9079 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 18:39:01.729972 9182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:01.730320 9079 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:01.730755 9079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 18:39:01.730882 9079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/wal
uuid: "9c9618bf0f8648ed987531e4f6839082"
format_stamp: "Formatted at 2026-09-27 18:39:01 on dist-test-slave-9jll"
server_key: "446c62010ac963436df4418276b8efaa"
server_key_iv: "fe9390d9755cce893d021a49f408dc79"
server_key_version: "encryptionkey@0"
I20260927 18:39:01.736344 9079 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:01.743667 9079 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 18:39:01.744014 9079 log.cc:1196] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 18:39:01.744561 9079 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: Deleting consensus metadata
I20260927 18:39:01.744870 9079 master_runner.cc:315] Copying system tablet from 127.2.95.126:39071
I20260927 18:39:01.747776 9079 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: tablet copy: Beginning tablet copy session from remote peer at address 127.2.95.126:39071
I20260927 18:39:01.765118 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.748863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.2.95.126:39071)
0927 18:39:01.749068 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:01.749075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:01.750223 (+ 1148us) server_negotiation.cc:407] Connection header received
0927 18:39:01.750525 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:01.750532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:01.750621 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:01.750738 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:01.751887 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.753033 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.754894 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.755576 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.755701 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.758274 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:01.758282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:01.758287 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:01.758331 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:01.761878 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.762819 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.762825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.762829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.762991 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:01.763447 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:01.763452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:01.763455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:01.763826 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:01.763988 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:01.764482 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 18:39:01.764719 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:39:01.765633 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:01.748824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:46588)
0927 18:39:01.750096 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:01.750122 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:01.750351 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:01.750784 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:01.750788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:01.751308 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:01.751643 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.751656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.753207 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.753212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.754702 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:01.754721 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:01.755760 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:01.755766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:01.755950 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:01.756764 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:01.756788 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:01.758048 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:01.762054 (+ 4006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.762061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.762065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.762674 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.763112 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:01.763116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:01.763120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:01.763324 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:01.763973 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:01.763979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:01.764190 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 18:39:01.764353 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 18:39:01.764747 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"thread_start_us":975,"threads_started":1}
I20260927 18:39:01.766867 8638 tablet_copy_service.cc:140] P 58a7477d2f2741d48e6705163e08facd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9c9618bf0f8648ed987531e4f6839082 ({username='oryx', principal='oryx/127.2.95.125@KRBTEST.COM'} at 127.0.0.1:46588)
I20260927 18:39:01.767109 8638 tablet_copy_service.cc:161] P 58a7477d2f2741d48e6705163e08facd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9c9618bf0f8648ed987531e4f6839082 at {username='oryx', principal='oryx/127.2.95.125@KRBTEST.COM'} at 127.0.0.1:46588: session id = 9c9618bf0f8648ed987531e4f6839082-00000000000000000000000000000000
I20260927 18:39:01.771747 8638 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 18:39:01.774370 9079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 18:39:01.781492 9079 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: tablet copy: Starting download of 0 data blocks...
I20260927 18:39:01.781883 9079 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: tablet copy: Starting download of 1 WAL segments...
I20260927 18:39:01.784931 9079 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 18:39:01.789903 8638 tablet_copy_service.cc:342] P 58a7477d2f2741d48e6705163e08facd: Request end of tablet copy session 9c9618bf0f8648ed987531e4f6839082-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.95.125@KRBTEST.COM'} at 127.0.0.1:46588
I20260927 18:39:01.790126 8638 tablet_copy_service.cc:433] P 58a7477d2f2741d48e6705163e08facd: ending tablet copy session 9c9618bf0f8648ed987531e4f6839082-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9c9618bf0f8648ed987531e4f6839082
I20260927 18:39:01.804672 9079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:01.807778 9189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:01.813005 9190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:01.823870 9192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:01.827306 9079 server_base.cc:1034] running on GCE node
I20260927 18:39:01.827754 9079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:01.828572 9079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:01.829797 9079 hybrid_clock.cc:648] HybridClock initialized: now 1790534341829752 us; error 60 us; skew 500 ppm
I20260927 18:39:01.830107 9079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:01 dist-test-slave-9jll krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534341, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:01.832479 9079 init.cc:377] Logged in from keytab as oryx/127.2.95.125@KRBTEST.COM (short username oryx)
I20260927 18:39:01.833881 9079 webserver.cc:460] Webserver started at http://127.2.95.125:41353/ using document root <none> and password file <none>
I20260927 18:39:01.834257 9079 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:01.834337 9079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:01.846556 9079 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.002s sys 0.010s
I20260927 18:39:01.848944 9199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:01.849550 9079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 18:39:01.849722 9079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/wal
uuid: "9c9618bf0f8648ed987531e4f6839082"
format_stamp: "Formatted at 2026-09-27 18:39:01 on dist-test-slave-9jll"
server_key: "446c62010ac963436df4418276b8efaa"
server_key_iv: "fe9390d9755cce893d021a49f408dc79"
server_key_version: "encryptionkey@0"
I20260927 18:39:01.849931 9079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:01.900051 9079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:01.904074 9079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:01.937978 9079 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.125:43979
I20260927 18:39:01.938025 9250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.125:43979 every 8 connection(s)
I20260927 18:39:01.939071 9079 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
I20260927 18:39:01.944554 9251 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:39:01.948761 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: Bootstrap starting.
I20260927 18:39:01.968659 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:39:01.969241 9251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: Bootstrap complete.
I20260927 18:39:01.970494 9251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } }
I20260927 18:39:01.970762 9251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9c9618bf0f8648ed987531e4f6839082, State: Initialized, Role: LEARNER
I20260927 18:39:01.971026 9251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } }
I20260927 18:39:01.971870 9254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } }
I20260927 18:39:01.972178 9251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:39:01.972339 9254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:39:01.982474 9265 catalog_manager.cc:1259] Loaded cluster ID: dac25e4c532440a4ba8372b8b7af6383
I20260927 18:39:01.982595 9265 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: loading cluster ID for follower catalog manager: success
I20260927 18:39:01.986344 9265 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: acquiring CA information for follower catalog manager: success
I20260927 18:39:01.989596 9265 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:39:02.047626 9074 heartbeater.cc:503] Master 127.2.95.126:39071 was elected leader, sending a full tablet report...
W20260927 18:39:02.110366 9179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:02.110685 9179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:02.110773 9179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:02.118669 9179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:39:02.118820 9179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:02.118906 9179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:02.118965 9179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:39:02.119010 9179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:39:02.129251 9179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:39071,127.2.95.125:43979,127.2.95.124:39795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.124.keytab
--principal=oryx/127.2.95.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.124:39795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.95.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:02.130699 9179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:02.132035 9179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:02.146302 9270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:02.146230 9273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:02.146273 9271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:02.147742 9179 server_base.cc:1034] running on GCE node
I20260927 18:39:02.148695 9179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:02.150427 9179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:02.151679 9179 hybrid_clock.cc:648] HybridClock initialized: now 1790534342151640 us; error 50 us; skew 500 ppm
I20260927 18:39:02.152117 9179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:02 dist-test-slave-9jll krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534342, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:02.155952 9179 init.cc:377] Logged in from keytab as oryx/127.2.95.124@KRBTEST.COM (short username oryx)
I20260927 18:39:02.158284 9179 webserver.cc:460] Webserver started at http://127.2.95.124:44203/ using document root <none> and password file <none>
I20260927 18:39:02.159042 9179 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:02.159158 9179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:02.159462 9179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:02.162744 9179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data/instance:
uuid: "346e593c3daa4b208ada0a83c81199e5"
format_stamp: "Formatted at 2026-09-27 18:39:02 on dist-test-slave-9jll"
server_key: "274ef5caa8aa48593a251ea2a995325b"
server_key_iv: "496637bdd1bf3ccdda0eb8ddf12bf0dd"
server_key_version: "encryptionkey@0"
I20260927 18:39:02.163551 9179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/wal/instance:
uuid: "346e593c3daa4b208ada0a83c81199e5"
format_stamp: "Formatted at 2026-09-27 18:39:02 on dist-test-slave-9jll"
server_key: "274ef5caa8aa48593a251ea2a995325b"
server_key_iv: "496637bdd1bf3ccdda0eb8ddf12bf0dd"
server_key_version: "encryptionkey@0"
I20260927 18:39:02.169134 9179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:39:02.173243 9280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:02.175071 9179 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:39:02.175261 9179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/wal
uuid: "346e593c3daa4b208ada0a83c81199e5"
format_stamp: "Formatted at 2026-09-27 18:39:02 on dist-test-slave-9jll"
server_key: "274ef5caa8aa48593a251ea2a995325b"
server_key_iv: "496637bdd1bf3ccdda0eb8ddf12bf0dd"
server_key_version: "encryptionkey@0"
I20260927 18:39:02.175477 9179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:02.208016 9179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:02.213320 9179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:02.213639 9179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:02.233573 9179 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.124:39795
I20260927 18:39:02.233605 9331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.124:39795 every 8 connection(s)
I20260927 18:39:02.235155 9179 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
I20260927 18:39:02.239727 9332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:02.240067 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 9179
I20260927 18:39:02.249991 9332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:39:02 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.95.124@KRBTEST.COM
Sep 27 18:39:02 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534342, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.124@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:02.265406 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.242450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.2.95.124:39795)
0927 18:39:02.243319 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:02.243336 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:02.243433 (+ 97us) server_negotiation.cc:407] Connection header received
0927 18:39:02.245491 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:02.245522 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:02.246079 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:02.246532 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:02.248188 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.249614 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.251064 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.251773 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.251974 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.256927 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:02.256939 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:02.256955 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:02.257020 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:02.260192 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.261109 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.261124 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.261134 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.261340 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.261882 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.261886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.261888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.262602 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:02.262874 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:02.263417 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 18:39:02.263896 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":392,"threads_started":1}
I20260927 18:39:02.270671 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.252853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.2.95.126:39071)
0927 18:39:02.253044 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:02.253049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:02.254132 (+ 1083us) server_negotiation.cc:407] Connection header received
0927 18:39:02.254346 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:02.254353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:02.254439 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:02.254538 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:02.255964 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.256791 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.258677 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.259101 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.259215 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.263154 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:02.263162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:02.263167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:02.263212 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:02.267215 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.268463 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.268469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.268473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.268650 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.269086 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.269091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.269094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.269416 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:02.269557 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:02.270104 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 18:39:02.270312 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":75}
I20260927 18:39:02.270712 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.252723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:46596)
0927 18:39:02.253974 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:02.254027 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:02.254191 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:02.254610 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:02.254615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:02.255408 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:02.255744 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.255763 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.257065 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.257069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.258519 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.258532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.259292 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.259303 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.259453 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.260078 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:02.260097 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:02.262925 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:02.267687 (+ 4762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.267697 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.267704 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.268327 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.268766 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.268771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.268774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.268964 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.269554 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:02.269562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:02.269778 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 18:39:02.269957 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 18:39:02.270312 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"thread_start_us":307,"threads_started":1}
I20260927 18:39:02.272112 9332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:39:02.275188 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.266974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:43979 (local address 127.0.0.1:36176)
0927 18:39:02.267731 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:02.267753 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:02.267877 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:02.269194 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:02.269199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:02.269231 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:02.269448 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.269457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.271114 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.271120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.272830 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.272848 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.274326 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.274332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.274527 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.274581 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 18:39:02.274670 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:39:02.274818 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":466,"threads_started":1}
I20260927 18:39:02.275204 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.267244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.2.95.125:43979)
0927 18:39:02.268798 (+ 1554us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:02.268804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:02.268823 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:39:02.268887 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:02.268892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:02.269001 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:02.269118 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:02.269589 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.270641 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.273056 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.274127 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.274272 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.274503 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:02.274709 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 18:39:02.274844 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1412,"thread_start_us":1298,"threads_started":1}
I20260927 18:39:02.280278 9226 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 18:39:02.280875 9254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a7477d2f2741d48e6705163e08facd. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } }
I20260927 18:39:02.281119 9254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: This master's current role is: LEARNER
Sep 27 18:39:02 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534342, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.124@KRBTEST.COM for oryx/127.2.95.125@KRBTEST.COM
I20260927 18:39:02.288030 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.272787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:43979 (local address 127.0.0.1:36192)
0927 18:39:02.272981 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:02.273000 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:02.273095 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:02.275713 (+ 2618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:02.275717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:02.276083 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:02.276232 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.276241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.277191 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.277194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.278330 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.278343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.279091 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.279097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.279288 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.279886 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:02.279903 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:02.282204 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:02.285631 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.285635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.285638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.286166 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.286606 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.286610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.286613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.286749 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.287294 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:02.287298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:02.287446 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 18:39:02.287555 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 18:39:02.287794 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 18:39:02.288204 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.272902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.2.95.125:43979)
0927 18:39:02.275448 (+ 2546us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:02.275453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:02.275469 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:39:02.275508 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:02.275511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:02.275569 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:02.275652 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:02.276362 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.277059 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.278467 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.278942 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.279084 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.282448 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:02.282454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:02.282457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:02.282491 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:02.285480 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.286301 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.286306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.286308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.286482 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.286892 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.286896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.286898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.287175 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:02.287294 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:02.287702 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 18:39:02.287899 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2441,"thread_start_us":6392,"threads_started":1}
I20260927 18:39:02.289222 9332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:39:02 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534342, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.124@KRBTEST.COM for oryx/127.2.95.124@KRBTEST.COM
I20260927 18:39:02.301481 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.290010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:39795 (local address 127.0.0.1:59122)
0927 18:39:02.290213 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:02.290228 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:02.290317 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:02.290557 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:02.290560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:02.290932 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:02.291090 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.291099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.292026 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.292030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.292907 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.292919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.293601 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.293606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.293800 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.294383 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:02.294400 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:02.296435 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:02.299228 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.299234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.299238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.299586 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.299889 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.299894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.299898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.300052 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.300550 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:02.300555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:02.300692 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 18:39:02.300804 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 18:39:02.301064 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 18:39:02.301518 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.290210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.2.95.124:39795)
0927 18:39:02.290353 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:02.290357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:02.290371 (+ 14us) server_negotiation.cc:407] Connection header received
0927 18:39:02.290410 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:02.290413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:02.290465 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:02.290539 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:02.291215 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.291894 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.293042 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.293437 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.293540 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.296643 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:02.296648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:02.296651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:02.296679 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:02.299098 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.299687 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.299691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.299695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.299797 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.300144 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.300149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.300152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.300443 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:02.300570 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:02.300893 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 18:39:02.301216 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":48}
I20260927 18:39:02.306504 9332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: Bootstrap starting.
I20260927 18:39:02.310088 9332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:02.311129 9332 log.cc:824] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:02.314054 9332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: No bootstrap required, opened a new log
I20260927 18:39:02.322443 9332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } }
I20260927 18:39:02.322887 9332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:02.322996 9332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346e593c3daa4b208ada0a83c81199e5, State: Initialized, Role: FOLLOWER
I20260927 18:39:02.323665 9332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } }
I20260927 18:39:02.324870 9342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } } }
I20260927 18:39:02.325389 9342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:39:02.326296 9332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:39:02.336355 9353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:39:02.336512 9353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:39:02.341732 9179 master_runner.cc:428] Detected that this master 346e593c3daa4b208ada0a83c81199e5 is joining an existing cluster
I20260927 18:39:02.341879 9179 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.95.124:39795
I20260927 18:39:02.372891 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 8679
I20260927 18:39:02.381359 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.347315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.2.95.126:39071)
0927 18:39:02.347521 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:02.347528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:02.349420 (+ 1892us) server_negotiation.cc:407] Connection header received
0927 18:39:02.349749 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:02.349755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:02.349860 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:02.349956 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:02.351913 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.353145 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.356195 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.356850 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.356981 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.373532 (+ 16551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:02.373540 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:02.373545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:02.373592 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:02.377537 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.378515 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.378521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.378524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.378693 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.379191 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.379196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.379200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.379689 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:02.379838 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:02.380735 (+ 897us) server_negotiation.cc:299] Negotiation successful
0927 18:39:02.380995 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:39:02.382514 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.347314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:46604)
0927 18:39:02.349310 (+ 1996us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:02.349335 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:02.349557 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:02.350181 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:02.350186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:02.351084 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:02.351680 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.351696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.353647 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.353652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.355897 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.355917 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.357245 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.357252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.357441 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.370021 (+ 12580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:02.370056 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:02.371614 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:02.377776 (+ 6162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.377782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.377786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.378332 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.378843 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.378848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.378852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.379039 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.379908 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:02.379913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:02.380135 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 18:39:02.380453 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 18:39:02.381840 (+ 1387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1850,"mutex_wait_us":81,"thread_start_us":1564,"threads_started":1}
I20260927 18:39:02.386518 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.349575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.2.95.125:43979)
0927 18:39:02.349768 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:02.349774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:02.354656 (+ 4882us) server_negotiation.cc:407] Connection header received
0927 18:39:02.354932 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:02.354939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:02.355023 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:02.355090 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:02.356635 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.357694 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.359193 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.359617 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.359728 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.372020 (+ 12292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:02.372029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:02.372034 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:02.372092 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:02.379841 (+ 7749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.380930 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.380936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.380940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.381116 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.381586 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.381590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.381593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.381993 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:02.382133 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:02.385875 (+ 3742us) server_negotiation.cc:299] Negotiation successful
0927 18:39:02.386167 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 18:39:02.387068 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.348137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59138 (local address 127.2.95.124:39795)
0927 18:39:02.348332 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:02.348337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:02.349568 (+ 1231us) server_negotiation.cc:407] Connection header received
0927 18:39:02.349805 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:02.349810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:02.349889 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:02.349955 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:02.351699 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.353444 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.355791 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.356505 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.356591 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.372116 (+ 15525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:02.372126 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:02.372131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:02.372162 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:02.377473 (+ 5311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.382201 (+ 4728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.382206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.382209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.382556 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.382985 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.382989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.382992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.384855 (+ 1863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:02.385032 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:02.386469 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0927 18:39:02.386690 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":69}
I20260927 18:39:02.390250 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.347369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:39795 (local address 127.0.0.1:59138)
0927 18:39:02.349453 (+ 2084us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:02.349471 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:02.349656 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:02.350161 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:02.350165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:02.350968 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:02.351501 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.351515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.353646 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.353651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.355632 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.355652 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.356910 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.356916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.357118 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.369996 (+ 12878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:02.370028 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:02.371933 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:02.380730 (+ 8797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.380734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.380738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.381190 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.382685 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.382689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.382692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.382884 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.385088 (+ 2204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:02.385092 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:02.386210 (+ 1118us) client_negotiation.cc:769] Sending connection context
0927 18:39:02.386936 (+ 726us) client_negotiation.cc:240] Negotiation successful
0927 18:39:02.387216 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1921,"mutex_wait_us":47,"spinlock_wait_cycles":29568,"thread_start_us":1269,"threads_started":1}
I20260927 18:39:02.391165 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.349287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:43979 (local address 127.0.0.1:36204)
0927 18:39:02.354547 (+ 5260us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:02.354568 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:02.354774 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:02.355291 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:02.355295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:02.356077 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:02.356456 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.356467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.357908 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.357913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.359047 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.359059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.359794 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.359800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.359965 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.370121 (+ 10156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:02.370140 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:02.371763 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:02.380169 (+ 8406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.380175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.380179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.380781 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.381280 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.381285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.381289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.381455 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.382214 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:02.382219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:02.385491 (+ 3272us) client_negotiation.cc:769] Sending connection context
0927 18:39:02.385733 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 18:39:02.387532 (+ 1799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5095,"mutex_wait_us":749,"thread_start_us":4243,"threads_started":1}
Sep 27 18:39:02 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.95.65@KRBTEST.COM
Sep 27 18:39:02 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534339, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.126@KRBTEST.COM for oryx/127.2.95.124@KRBTEST.COM
I20260927 18:39:02.404130 9362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.374825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51600 (local address 127.2.95.65:39523)
0927 18:39:02.383378 (+ 8553us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:02.383386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:02.383411 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:39:02.383466 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:02.383473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:02.383745 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:02.383890 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:02.385536 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.387318 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.389088 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.390175 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.392913 (+ 2738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.397478 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:02.397501 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:02.397523 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:02.397580 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:02.400813 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.401564 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.401575 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.401578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.401765 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.402136 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.402141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.402145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.402563 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:02.402892 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:02.403459 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 18:39:02.403751 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8386,"thread_start_us":8221,"threads_started":1}
I20260927 18:39:02.405336 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.389154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:39795 (local address 127.0.0.1:59148)
0927 18:39:02.389359 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:02.389379 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:02.389485 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:02.390203 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:02.390209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:02.390678 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:02.390867 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.390879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.392468 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.392475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.393957 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.393976 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.395043 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.395050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.395245 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.396261 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:02.396285 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:02.398534 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:02.402435 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.402441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.402445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.403001 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.403446 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.403452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.403456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.403634 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.404373 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:02.404378 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:02.404541 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 18:39:02.404686 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 18:39:02.404971 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 18:39:02.405575 8802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:02.406764 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.389530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.2.95.124:39795)
0927 18:39:02.389724 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:02.389730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:02.389748 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:39:02.389863 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:02.389869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:02.390022 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:02.391027 (+ 1005us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:02.391205 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.392297 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.394183 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.394845 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.394977 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.398774 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:02.398781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:02.398786 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:02.398837 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:02.402265 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.403131 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.403136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.403140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.403287 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.403770 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.403775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.403778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.404145 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:02.404323 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:02.405811 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0927 18:39:02.406050 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:39:02.408237 8618 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.95.124:39795: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "346e593c3daa4b208ada0a83c81199e5"
member_type: NON_VOTER
last_known_addr {
host: "127.2.95.124"
port: 39795
}
attrs {
promote: true
}
}
dest_uuid: "58a7477d2f2741d48e6705163e08facd"
cas_config_opid_index: 5
I20260927 18:39:02.409207 8618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } }
I20260927 18:39:02.410823 9337 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 346e593c3daa4b208ada0a83c81199e5 (127.2.95.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } } }
I20260927 18:39:02.415764 9226 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 LEARNER]: Refusing update from remote peer 58a7477d2f2741d48e6705163e08facd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 18:39:02.417443 9307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:02.421384 9307 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 FOLLOWER]: Refusing update from remote peer 58a7477d2f2741d48e6705163e08facd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 18:39:02.426848 9364 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 18:39:02.432226 9226 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 346e593c3daa4b208ada0a83c81199e5 (127.2.95.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } } }
I20260927 18:39:02.435374 9366 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [LEADER]: Connected to new peer: Peer: permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:02.437302 9254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } } }
I20260927 18:39:02.437592 9254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:39:02.443835 9337 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.95.124:39795
I20260927 18:39:02.444950 9337 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd: attempting to promote NON_VOTER 9c9618bf0f8648ed987531e4f6839082 to VOTER
I20260927 18:39:02.444741 9365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } } }
I20260927 18:39:02.445212 9365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:02.449352 9179 master.cc:537] Master@127.2.95.124:39795 shutting down...
I20260927 18:39:02.451210 9337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } }
I20260927 18:39:02.451812 9307 raft_consensus.cc:858] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } }
W20260927 18:39:02.452023 9307 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } }
I20260927 18:39:02.455327 9226 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 LEARNER]: Refusing update from remote peer 58a7477d2f2741d48e6705163e08facd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 18:39:02.458844 9365 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
W20260927 18:39:02.466187 9286 tablet.cc:2369] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:39:02.467417 8602 consensus_peers.cc:487] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd -> Peer 346e593c3daa4b208ada0a83c81199e5 (127.2.95.124:39795): Couldn't send request to peer 346e593c3daa4b208ada0a83c81199e5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 18:39:02.468247 9368 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9c9618bf0f8648ed987531e4f6839082 (127.2.95.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } } }
I20260927 18:39:02.472733 9337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } } }
I20260927 18:39:02.473038 9337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:02.473392 9337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } } }
I20260927 18:39:02.473596 9337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:02.473501 9226 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9c9618bf0f8648ed987531e4f6839082 (127.2.95.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } } }
I20260927 18:39:02.477895 9254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } } }
I20260927 18:39:02.478242 9254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:39:02.507331 9307 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9c9618bf0f8648ed987531e4f6839082 (127.2.95.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } }
I20260927 18:39:02.526176 9376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } }
I20260927 18:39:02.526536 9376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 18:39:02.526598 9342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a7477d2f2741d48e6705163e08facd. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } }
I20260927 18:39:02.526844 9342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 18:39:02.527187 9373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: NON_VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: true } } }
I20260927 18:39:02.527420 9373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 18:39:02.529759 9179 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 18:39:02.530536 9179 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 18:39:02.530774 9179 tablet_replica.cc:330] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: stopping tablet replica
I20260927 18:39:02.548821 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 8679
I20260927 18:39:02.549150 9179 master.cc:552] Master@127.2.95.124:39795 shutdown complete.
I20260927 18:39:02.549232 9179 master_runner.cc:305] Clearing existing system tablet
I20260927 18:39:02.549584 9179 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:02.549707 9179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:02.552246 9179 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 18:39:02.553987 9378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:02.554382 9179 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:02.554637 9179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:39:02.554735 9179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/wal
uuid: "346e593c3daa4b208ada0a83c81199e5"
format_stamp: "Formatted at 2026-09-27 18:39:02 on dist-test-slave-9jll"
server_key: "274ef5caa8aa48593a251ea2a995325b"
server_key_iv: "496637bdd1bf3ccdda0eb8ddf12bf0dd"
server_key_version: "encryptionkey@0"
I20260927 18:39:02.558403 9179 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:02.565959 9179 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 18:39:02.566169 9179 log.cc:1196] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 18:39:02.566677 9179 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: Deleting consensus metadata
I20260927 18:39:02.567000 9179 master_runner.cc:315] Copying system tablet from 127.2.95.126:39071
I20260927 18:39:02.569893 9179 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: tablet copy: Beginning tablet copy session from remote peer at address 127.2.95.126:39071
I20260927 18:39:02.571398 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:39523
--local_ip_for_outbound_sockets=127.2.95.65
--tserver_master_addrs=127.2.95.126:39071
--webserver_port=34901
--webserver_interface=127.2.95.65
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab
--principal=oryx/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.95.126:39071,127.2.95.125:43979
--tserver_master_addrs=127.2.95.126:39071,127.2.95.125:43979,127.2.95.124:39795 with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:39:02.585460 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.574094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.2.95.126:39071)
0927 18:39:02.574300 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:02.574304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:02.574322 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:39:02.574387 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:02.574391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:02.574464 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:02.574568 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:02.575449 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.576191 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.577481 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.577939 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.578053 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.580225 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:02.580231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:02.580234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:02.580267 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:02.582854 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.583532 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.583537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.583541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.583668 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:02.584078 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:02.584082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:02.584086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:02.584365 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:02.584496 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:02.584899 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 18:39:02.585118 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 18:39:02.585556 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:02.570665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:46614)
0927 18:39:02.571179 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:02.571195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:02.571313 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:02.574628 (+ 3315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:02.574633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:02.575049 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:02.575251 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.575262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.576336 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.576340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.577333 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:02.577345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:02.578103 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:02.578109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:02.578265 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:02.578975 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:02.578997 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:02.580024 (+ 1027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:02.582996 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.583002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.583007 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.583402 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.583792 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:02.583798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:02.583802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:02.583972 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:02.584484 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:02.584488 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:02.584651 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 18:39:02.584770 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 18:39:02.585105 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":270,"threads_started":1}
I20260927 18:39:02.586501 8638 tablet_copy_service.cc:140] P 58a7477d2f2741d48e6705163e08facd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 346e593c3daa4b208ada0a83c81199e5 ({username='oryx', principal='oryx/127.2.95.124@KRBTEST.COM'} at 127.0.0.1:46614)
I20260927 18:39:02.586650 8638 tablet_copy_service.cc:161] P 58a7477d2f2741d48e6705163e08facd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 346e593c3daa4b208ada0a83c81199e5 at {username='oryx', principal='oryx/127.2.95.124@KRBTEST.COM'} at 127.0.0.1:46614: session id = 346e593c3daa4b208ada0a83c81199e5-00000000000000000000000000000000
I20260927 18:39:02.588869 8638 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 18:39:02.590552 9179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 18:39:02.595573 9179 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: tablet copy: Starting download of 0 data blocks...
I20260927 18:39:02.595847 9179 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: tablet copy: Starting download of 1 WAL segments...
I20260927 18:39:02.598052 9179 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 18:39:02.601553 8638 tablet_copy_service.cc:342] P 58a7477d2f2741d48e6705163e08facd: Request end of tablet copy session 346e593c3daa4b208ada0a83c81199e5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.95.124@KRBTEST.COM'} at 127.0.0.1:46614
I20260927 18:39:02.601712 8638 tablet_copy_service.cc:433] P 58a7477d2f2741d48e6705163e08facd: ending tablet copy session 346e593c3daa4b208ada0a83c81199e5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 346e593c3daa4b208ada0a83c81199e5
I20260927 18:39:02.613008 9179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:02.617844 9386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:02.618805 9387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:02.626051 9389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:02.634302 9179 server_base.cc:1034] running on GCE node
I20260927 18:39:02.634701 9179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:02.635469 9179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:02.636706 9179 hybrid_clock.cc:648] HybridClock initialized: now 1790534342636654 us; error 40 us; skew 500 ppm
I20260927 18:39:02.636991 9179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:02 dist-test-slave-9jll krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534342, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:02.639281 9179 init.cc:377] Logged in from keytab as oryx/127.2.95.124@KRBTEST.COM (short username oryx)
I20260927 18:39:02.642727 9179 webserver.cc:460] Webserver started at http://127.2.95.124:41935/ using document root <none> and password file <none>
I20260927 18:39:02.643052 9179 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:02.643147 9179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:02.646278 9179 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260927 18:39:02.648332 9396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:02.648746 9179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 18:39:02.648938 9179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/wal
uuid: "346e593c3daa4b208ada0a83c81199e5"
format_stamp: "Formatted at 2026-09-27 18:39:02 on dist-test-slave-9jll"
server_key: "274ef5caa8aa48593a251ea2a995325b"
server_key_iv: "496637bdd1bf3ccdda0eb8ddf12bf0dd"
server_key_version: "encryptionkey@0"
I20260927 18:39:02.649202 9179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:02.674589 9179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:02.678961 9179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:02.705178 9179 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.124:39795
I20260927 18:39:02.705238 9447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.124:39795 every 8 connection(s)
I20260927 18:39:02.706414 9179 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
I20260927 18:39:02.712014 9448 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:39:02.715842 9448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: Bootstrap starting.
I20260927 18:39:02.731336 9448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:39:02.731905 9448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: Bootstrap complete.
I20260927 18:39:02.732999 9448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } }
I20260927 18:39:02.733295 9448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 346e593c3daa4b208ada0a83c81199e5, State: Initialized, Role: LEARNER
I20260927 18:39:02.733553 9448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } }
I20260927 18:39:02.734407 9451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } } }
I20260927 18:39:02.734716 9448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:39:02.734928 9451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:39:02.744536 9462 catalog_manager.cc:1259] Loaded cluster ID: dac25e4c532440a4ba8372b8b7af6383
I20260927 18:39:02.744660 9462 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: loading cluster ID for follower catalog manager: success
I20260927 18:39:02.748394 9462 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: acquiring CA information for follower catalog manager: success
I20260927 18:39:02.751048 9462 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 18:39:02.949179 9385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:02.949524 9385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:02.949662 9385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:02.957850 9385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:02.958005 9385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:02.958165 9385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:39:02.968237 9385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab
--principal=oryx/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:39523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=34901
--tserver_master_addrs=127.2.95.126:39071,127.2.95.125:43979,127.2.95.124:39795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:02.969923 9385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:02.971369 9385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:02.985961 9467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:02.985896 9468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:02.986140 9470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:02.986855 9385 server_base.cc:1034] running on GCE node
I20260927 18:39:02.987558 9385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:02.988582 9385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:02.989786 9385 hybrid_clock.cc:648] HybridClock initialized: now 1790534342989716 us; error 65 us; skew 500 ppm
I20260927 18:39:02.990136 9385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:02 dist-test-slave-9jll krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534342, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:02.993573 9385 init.cc:377] Logged in from keytab as oryx/127.2.95.65@KRBTEST.COM (short username oryx)
I20260927 18:39:02.995127 9385 webserver.cc:460] Webserver started at http://127.2.95.65:34901/ using document root <none> and password file <none>
I20260927 18:39:02.995828 9385 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:02.995965 9385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:03.001739 9385 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 18:39:03.005427 9477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:03.007303 9385 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:39:03.007503 9385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "52c5acedd2a3495c83ed550b02b06660"
format_stamp: "Formatted at 2026-09-27 18:38:59 on dist-test-slave-9jll"
server_key: "279f84570e30cb1d770ed8867298d203"
server_key_iv: "c3fe08bd56a511a6b48640fa2f612655"
server_key_version: "encryptionkey@0"
I20260927 18:39:03.008759 9385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:03.042781 9385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:03.047286 9385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:03.047564 9385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:03.048678 9385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:03.050328 9385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:03.050439 9385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:03.050544 9385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:03.050631 9385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:03.078610 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.071266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:39795 (local address 127.0.0.1:59154)
0927 18:39:03.071657 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:03.071679 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:03.071822 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:03.072734 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:03.072740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:03.072793 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:03.073034 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.073047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.074649 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.074656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.076282 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.076303 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.077986 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.077993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.078170 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.078198 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 18:39:03.078268 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 18:39:03.078374 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":191,"threads_started":1}
I20260927 18:39:03.078841 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.071427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.2.95.124:39795)
0927 18:39:03.072344 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:03.072350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:03.072370 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:39:03.072431 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:03.072435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:03.072514 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:03.072685 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:03.073175 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.074452 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.076461 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.077765 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.077926 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.078227 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:03.078301 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 18:39:03.078455 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":627,"threads_started":1}
I20260927 18:39:03.079638 9423 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 18:39:03.080102 9451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a7477d2f2741d48e6705163e08facd. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } } }
I20260927 18:39:03.080363 9451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: This master's current role is: LEARNER
I20260927 18:39:03.089177 9385 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:39523
I20260927 18:39:03.089265 9591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:39523 every 8 connection(s)
I20260927 18:39:03.090646 9385 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:39:03.099958 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 9385
I20260927 18:39:03.100296 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 8813
Sep 27 18:39:03 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534342, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.125@KRBTEST.COM
Sep 27 18:39:03 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534342, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.124@KRBTEST.COM
Sep 27 18:39:03 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534342, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:03.122207 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.093835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:51471 (local address 127.2.95.124:39795)
0927 18:39:03.094052 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:03.094059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:03.096130 (+ 2071us) server_negotiation.cc:407] Connection header received
0927 18:39:03.098590 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:03.098596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:03.098672 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:03.098748 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:03.101224 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.102380 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.105920 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.106564 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.106688 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.113705 (+ 7017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:03.113713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:03.113718 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:03.113763 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:03.118003 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.119037 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.119043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.119046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.119237 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.119821 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.119826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.119829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.120181 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:03.120303 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:03.121571 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0927 18:39:03.121878 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:39:03.122756 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.093756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:36171 (local address 127.2.95.125:43979)
0927 18:39:03.094440 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:03.094444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:03.096128 (+ 1684us) server_negotiation.cc:407] Connection header received
0927 18:39:03.098505 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:03.098510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:03.098622 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:03.098702 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:03.101224 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.102379 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.105920 (+ 3541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.106576 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.106678 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.112578 (+ 5900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:03.112589 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:03.112619 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:03.112676 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:03.118248 (+ 5572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.120197 (+ 1949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.120203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.120206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.120379 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.121186 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.121191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.121195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.121527 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:03.121651 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:03.122194 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 18:39:03.122416 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":482,"threads_started":1}
I20260927 18:39:03.123250 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.096471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.65:44707)
0927 18:39:03.097027 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:03.097045 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:03.098210 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:03.098984 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:03.098993 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:03.099922 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:03.101014 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.101040 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.102581 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.102587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.105732 (+ 3145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.105750 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.107145 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.107156 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.107468 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.108385 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:03.108412 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:03.113402 (+ 4990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:03.118043 (+ 4641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:03.118055 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:03.118074 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:03.118897 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:03.119352 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:03.119357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:03.119361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:03.119674 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:03.120355 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:03.120364 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:03.120894 (+ 530us) client_negotiation.cc:769] Sending connection context
0927 18:39:03.121218 (+ 324us) client_negotiation.cc:240] Negotiation successful
0927 18:39:03.121798 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":374,"spinlock_wait_cycles":44032,"thread_start_us":257,"threads_started":1}
I20260927 18:39:03.123700 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.094819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:43979 (local address 127.2.95.65:36171)
0927 18:39:03.095937 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:03.096008 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:03.098205 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:03.098985 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:03.098995 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:03.099965 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:03.101006 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.101035 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.102581 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.102586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.105732 (+ 3146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.105749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.107059 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.107067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.107460 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.108303 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:03.108344 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:03.111865 (+ 3521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:03.118394 (+ 6529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:03.118398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:03.118402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:03.120052 (+ 1650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:03.120517 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:03.120521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:03.120524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:03.121048 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:03.121751 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:03.121755 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:03.121905 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 18:39:03.122956 (+ 1051us) client_negotiation.cc:240] Negotiation successful
0927 18:39:03.123259 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":1487,"spinlock_wait_cycles":11904,"thread_start_us":404,"threads_started":1}
I20260927 18:39:03.124145 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.094819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:39795 (local address 127.2.95.65:51471)
0927 18:39:03.095935 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:03.096008 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:03.098205 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:03.098985 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:03.098995 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:03.099921 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:03.101006 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.101035 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.102581 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.102586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.105734 (+ 3148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.105749 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.107059 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.107066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.107444 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.108507 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:03.108526 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:03.111903 (+ 3377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:03.118171 (+ 6268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:03.118177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:03.118181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:03.118880 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:03.119367 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:03.119372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:03.119375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:03.119684 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:03.120360 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:03.120370 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:03.120894 (+ 524us) client_negotiation.cc:769] Sending connection context
0927 18:39:03.121217 (+ 323us) client_negotiation.cc:240] Negotiation successful
0927 18:39:03.123613 (+ 2396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":530,"spinlock_wait_cycles":155776,"thread_start_us":388,"threads_started":1}
I20260927 18:39:03.125190 9594 heartbeater.cc:347] Connected to a master server at 127.2.95.125:43979
I20260927 18:39:03.125540 9594 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:03.125820 9592 heartbeater.cc:347] Connected to a master server at 127.2.95.124:39795
I20260927 18:39:03.126092 9592 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:03.126106 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.096595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:44707 (local address 127.2.95.126:39071)
0927 18:39:03.097245 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:03.097249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:03.097265 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:39:03.098526 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:03.098532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:03.098624 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:03.098721 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:03.101227 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.102363 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.105940 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.106732 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.106864 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.113800 (+ 6936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:03.113808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:03.113813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:03.113862 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:03.117848 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.119053 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.119059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.119062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.119221 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.119811 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.119816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.119820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.120153 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:03.120288 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:03.124339 (+ 4051us) server_negotiation.cc:299] Negotiation successful
0927 18:39:03.125727 (+ 1388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":457,"threads_started":1}
I20260927 18:39:03.126634 9594 heartbeater.cc:511] Master 127.2.95.125:43979 requested a full tablet report, sending...
I20260927 18:39:03.126955 9593 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39071
Sep 27 18:39:03 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.95.66@KRBTEST.COM
I20260927 18:39:03.127076 9593 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:03.127545 9592 heartbeater.cc:511] Master 127.2.95.124:39795 requested a full tablet report, sending...
I20260927 18:39:03.128991 9216 ts_manager.cc:194] Registered new tserver with Master: 52c5acedd2a3495c83ed550b02b06660 (127.2.95.65:39523)
I20260927 18:39:03.129227 9413 ts_manager.cc:194] Registered new tserver with Master: 52c5acedd2a3495c83ed550b02b06660 (127.2.95.65:39523)
I20260927 18:39:03.129462 9593 heartbeater.cc:511] Master 127.2.95.126:39071 requested a full tablet report, sending...
I20260927 18:39:03.130625 8618 ts_manager.cc:194] Re-registered known tserver with Master: 52c5acedd2a3495c83ed550b02b06660 (127.2.95.65:39523)
I20260927 18:39:03.131451 8618 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:44707
I20260927 18:39:03.136157 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.101631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.2.95.66:36525)
0927 18:39:03.114522 (+ 12891us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:03.114532 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:03.114567 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:39:03.114633 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:03.114642 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:03.114974 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:03.115145 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:03.117442 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.119005 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.120828 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.121695 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.121815 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.128223 (+ 6408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:03.128244 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:03.128261 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:03.128309 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:03.133207 (+ 4898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.134091 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.134098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.134100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.134237 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.134589 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.134593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.134595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.134952 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:03.135229 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:03.135553 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 18:39:03.135807 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12687,"thread_start_us":13683,"threads_started":1}
I20260927 18:39:03.137378 8936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:03.221052 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 8813
I20260927 18:39:03.234225 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:36525
--local_ip_for_outbound_sockets=127.2.95.66
--tserver_master_addrs=127.2.95.126:39071
--webserver_port=35063
--webserver_interface=127.2.95.66
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab
--principal=oryx/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.95.126:39071,127.2.95.125:43979
--tserver_master_addrs=127.2.95.126:39071,127.2.95.125:43979,127.2.95.124:39795 with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:39:03.521062 9368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: NON_VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: true } } }
I20260927 18:39:03.521521 9368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:03.524392 9605 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd: attempting to promote NON_VOTER 346e593c3daa4b208ada0a83c81199e5 to VOTER
I20260927 18:39:03.525028 9605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } }
I20260927 18:39:03.527096 9423 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 LEARNER]: Refusing update from remote peer 58a7477d2f2741d48e6705163e08facd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 18:39:03.527096 9226 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 FOLLOWER]: Refusing update from remote peer 58a7477d2f2741d48e6705163e08facd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 18:39:03.527674 9368 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 18:39:03.528021 9606 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [LEADER]: Connected to new peer: Peer: permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 18:39:03.531615 9226 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 346e593c3daa4b208ada0a83c81199e5 (127.2.95.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } } }
I20260927 18:39:03.531670 9423 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 346e593c3daa4b208ada0a83c81199e5 (127.2.95.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } } }
I20260927 18:39:03.530988 9605 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 346e593c3daa4b208ada0a83c81199e5 (127.2.95.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } } }
I20260927 18:39:03.535378 9451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } } }
I20260927 18:39:03.535660 9451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:39:03.537738 9606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } } }
I20260927 18:39:03.538064 9606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:03.538383 9605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } } }
I20260927 18:39:03.538672 9605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:03.538488 9606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } } }
I20260927 18:39:03.538794 9606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:03.543128 9611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58a7477d2f2741d48e6705163e08facd" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } } }
I20260927 18:39:03.543541 9611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 18:39:03.572672 9604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:03.573000 9604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:03.573122 9604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:03.580927 9604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:03.581048 9604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:03.581163 9604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:39:03.591480 9604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab
--principal=oryx/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:36525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=35063
--tserver_master_addrs=127.2.95.126:39071,127.2.95.125:43979,127.2.95.124:39795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:03.592730 9604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:03.594110 9604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:03.607810 9617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:03.607858 9616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:03.607894 9619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:03.608238 9604 server_base.cc:1034] running on GCE node
I20260927 18:39:03.608982 9604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:03.610119 9604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:03.611356 9604 hybrid_clock.cc:648] HybridClock initialized: now 1790534343611296 us; error 54 us; skew 500 ppm
I20260927 18:39:03.611711 9604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:03 dist-test-slave-9jll krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534343, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:03.615265 9604 init.cc:377] Logged in from keytab as oryx/127.2.95.66@KRBTEST.COM (short username oryx)
I20260927 18:39:03.616837 9604 webserver.cc:460] Webserver started at http://127.2.95.66:35063/ using document root <none> and password file <none>
I20260927 18:39:03.617537 9604 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:03.617651 9604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:03.623420 9604 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:39:03.626226 9626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:03.627722 9604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 18:39:03.627990 9604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36"
format_stamp: "Formatted at 2026-09-27 18:39:00 on dist-test-slave-9jll"
server_key: "0fa6d7cda388e3ed663fcd38d5a24941"
server_key_iv: "1472fc326598cf20c385e8e196003297"
server_key_version: "encryptionkey@0"
I20260927 18:39:03.628959 9604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:03.644492 9604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:03.648860 9604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:03.649137 9604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:03.650297 9604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:03.651770 9604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:03.651866 9604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:03.651976 9604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:03.652043 9604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:03.684391 9604 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:36525
I20260927 18:39:03.684450 9738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:36525 every 8 connection(s)
I20260927 18:39:03.685899 9604 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:39:03.686951 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 9604
I20260927 18:39:03.687266 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 8946
Sep 27 18:39:03 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534343, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
Sep 27 18:39:03 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534343, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.125@KRBTEST.COM
Sep 27 18:39:03 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534343, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.124@KRBTEST.COM
Sep 27 18:39:03 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.95.67@KRBTEST.COM
I20260927 18:39:03.726679 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.690743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:42475 (local address 127.2.95.126:39071)
0927 18:39:03.698135 (+ 7392us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:03.698143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:03.698172 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:39:03.698249 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:03.698255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:03.698359 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:03.698516 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:03.702199 (+ 3683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.703510 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.709793 (+ 6283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.710445 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.710568 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.718413 (+ 7845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:03.718421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:03.718426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:03.718470 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:03.723222 (+ 4752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.723943 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.723948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.723951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.724093 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.724503 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.724507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.724510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.724819 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:03.724946 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:03.726122 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0927 18:39:03.726339 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7259,"thread_start_us":7224,"threads_started":1}
I20260927 18:39:03.727730 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.705590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:43979 (local address 127.2.95.66:46435)
0927 18:39:03.706410 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:03.706431 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:03.706586 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:03.708245 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:03.708250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:03.708870 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:03.709221 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.709234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.710713 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.710719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.712318 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.712335 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.713541 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.713547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.713766 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.715680 (+ 1914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:03.715704 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:03.718923 (+ 3219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:03.722732 (+ 3809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:03.722743 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:03.722765 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:03.723471 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:03.723913 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:03.723917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:03.723920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:03.724169 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:03.724903 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:03.724912 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:03.725389 (+ 477us) client_negotiation.cc:769] Sending connection context
0927 18:39:03.725753 (+ 364us) client_negotiation.cc:240] Negotiation successful
0927 18:39:03.726319 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":7,"spinlock_wait_cycles":4736,"thread_start_us":616,"threads_started":1}
I20260927 18:39:03.728135 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.690378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:39795 (local address 127.2.95.66:51215)
0927 18:39:03.699181 (+ 8803us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:03.699205 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:03.699338 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:03.706762 (+ 7424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:03.706767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:03.707476 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:03.709975 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.709986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.712350 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.712355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.714402 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.714420 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.715510 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.715516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.715695 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.716651 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:03.716674 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:03.720590 (+ 3916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:03.725231 (+ 4641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:03.725236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:03.725240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:03.725839 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:03.726268 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:03.726272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:03.726276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:03.726411 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:03.727167 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:03.727172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:03.727317 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 18:39:03.727454 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 18:39:03.727736 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8638,"mutex_wait_us":2136,"thread_start_us":2287,"threads_started":1}
I20260927 18:39:03.728184 9745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.689994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:51215 (local address 127.2.95.124:39795)
0927 18:39:03.706149 (+ 16155us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:03.706155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:03.706177 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:39:03.706235 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:03.706240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:03.706336 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:03.706459 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:03.710175 (+ 3716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.712175 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.714580 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.715265 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.715391 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.720823 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:03.720831 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:03.720835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:03.720904 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:03.725063 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.725993 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.725999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.726003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.726140 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.726535 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.726540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.726543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.727035 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:03.727185 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:03.727583 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 18:39:03.727790 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15993,"thread_start_us":15764,"threads_started":1}
I20260927 18:39:03.728983 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.690006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.66:42475)
0927 18:39:03.693176 (+ 3170us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:03.693239 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:03.696059 (+ 2820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:03.699845 (+ 3786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:03.699860 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:03.700872 (+ 1012us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:03.701880 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.701913 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.707177 (+ 5264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.707183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.709564 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:03.709578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.710906 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.710917 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:03.711348 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.713917 (+ 2569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:03.713954 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:03.717280 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:03.723359 (+ 6079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:03.723365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:03.723369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:03.723794 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:03.724223 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:03.724227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:03.724231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:03.724375 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:03.725004 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:03.725008 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:03.725389 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 18:39:03.725753 (+ 364us) client_negotiation.cc:240] Negotiation successful
0927 18:39:03.728620 (+ 2867us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2801,"mutex_wait_us":2342,"spinlock_wait_cycles":52608,"thread_start_us":8664,"threads_started":1}
I20260927 18:39:03.729883 9740 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39071
I20260927 18:39:03.730206 9740 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:03.730883 9739 heartbeater.cc:347] Connected to a master server at 127.2.95.124:39795
I20260927 18:39:03.730962 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.699643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:46435 (local address 127.2.95.125:43979)
0927 18:39:03.707755 (+ 8112us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:03.707761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:03.707779 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:39:03.707836 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:03.707840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:03.707950 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:03.708051 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:03.709413 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.710547 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.712502 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.713148 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.713260 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.719160 (+ 5900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:03.719167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:03.719171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:03.719223 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:03.722572 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.723617 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.723624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.723628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.723772 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.724296 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.724301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.724305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.724683 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:03.724828 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:03.730049 (+ 5221us) server_negotiation.cc:299] Negotiation successful
0927 18:39:03.730245 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7189,"thread_start_us":7059,"threads_started":1}
I20260927 18:39:03.731024 9739 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:03.731148 9740 heartbeater.cc:511] Master 127.2.95.126:39071 requested a full tablet report, sending...
I20260927 18:39:03.731438 9739 heartbeater.cc:511] Master 127.2.95.124:39795 requested a full tablet report, sending...
I20260927 18:39:03.731832 9741 heartbeater.cc:347] Connected to a master server at 127.2.95.125:43979
I20260927 18:39:03.731961 9741 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:03.732360 9741 heartbeater.cc:511] Master 127.2.95.125:43979 requested a full tablet report, sending...
I20260927 18:39:03.732832 8618 ts_manager.cc:194] Re-registered known tserver with Master: ba8c6a10b9e94dc0bea3e8e25c261e36 (127.2.95.66:36525)
I20260927 18:39:03.733147 9413 ts_manager.cc:194] Registered new tserver with Master: ba8c6a10b9e94dc0bea3e8e25c261e36 (127.2.95.66:36525)
I20260927 18:39:03.733453 9216 ts_manager.cc:194] Registered new tserver with Master: ba8c6a10b9e94dc0bea3e8e25c261e36 (127.2.95.66:36525)
I20260927 18:39:03.734000 8618 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:42475
I20260927 18:39:03.735412 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:03.688351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.2.95.67:37053)
0927 18:39:03.692084 (+ 3733us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:03.692095 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:03.704276 (+ 12181us) server_negotiation.cc:407] Connection header received
0927 18:39:03.707051 (+ 2775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:03.707061 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:03.707338 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:03.707489 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:03.711908 (+ 4419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.713518 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.717111 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:03.718055 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:03.718158 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:03.726756 (+ 8598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:03.726775 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:03.726793 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:03.726834 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:03.730710 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.732835 (+ 2125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.732845 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.732848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.733016 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:03.733613 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:03.733619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:03.733622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:03.734080 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:03.734404 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:03.734742 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 18:39:03.735019 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3489,"thread_start_us":3354,"threads_started":1}
I20260927 18:39:03.736804 9069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:03.821626 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 8946
I20260927 18:39:03.835220 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:37053
--local_ip_for_outbound_sockets=127.2.95.67
--tserver_master_addrs=127.2.95.126:39071
--webserver_port=38051
--webserver_interface=127.2.95.67
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab
--principal=oryx/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.95.126:39071,127.2.95.125:43979
--tserver_master_addrs=127.2.95.126:39071,127.2.95.125:43979,127.2.95.124:39795 with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:39:04.133694 9593 heartbeater.cc:503] Master 127.2.95.126:39071 was elected leader, sending a full tablet report...
W20260927 18:39:04.174713 9752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:04.175020 9752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:04.175119 9752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:04.182924 9752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:04.183068 9752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:04.183234 9752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:39:04.193539 9752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab
--principal=oryx/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:37053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=38051
--tserver_master_addrs=127.2.95.126:39071,127.2.95.125:43979,127.2.95.124:39795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:04.194998 9752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:04.196451 9752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:04.210525 9760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:04.210846 9757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:04.210505 9758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:04.211365 9752 server_base.cc:1034] running on GCE node
I20260927 18:39:04.212110 9752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:04.213214 9752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:04.214429 9752 hybrid_clock.cc:648] HybridClock initialized: now 1790534344214359 us; error 63 us; skew 500 ppm
I20260927 18:39:04.214738 9752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:04 dist-test-slave-9jll krb5kdc[8573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534344, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:04.218122 9752 init.cc:377] Logged in from keytab as oryx/127.2.95.67@KRBTEST.COM (short username oryx)
I20260927 18:39:04.219746 9752 webserver.cc:460] Webserver started at http://127.2.95.67:38051/ using document root <none> and password file <none>
I20260927 18:39:04.220460 9752 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:04.220582 9752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:04.226491 9752 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 18:39:04.229625 9767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:04.231351 9752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:39:04.231557 9752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "db6625f555d64f52b480f453712da79a"
format_stamp: "Formatted at 2026-09-27 18:39:00 on dist-test-slave-9jll"
server_key: "7e889087967ffc9319b5e9c13100acc7"
server_key_iv: "cd315c8a6aa6056d49ee38fea029aea0"
server_key_version: "encryptionkey@0"
I20260927 18:39:04.232600 9752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:04.266999 9752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:04.271623 9752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:04.271888 9752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:04.272887 9752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:04.274286 9752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:04.274371 9752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:04.274472 9752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:04.274545 9752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:04.305200 9752 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:37053
I20260927 18:39:04.305253 9879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:37053 every 8 connection(s)
I20260927 18:39:04.306740 9752 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:39:04.308755 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 9752
Sep 27 18:39:04 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534344, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.125@KRBTEST.COM
Sep 27 18:39:04 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534344, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
Sep 27 18:39:04 dist-test-slave-9jll krb5kdc[8573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534344, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.124@KRBTEST.COM
I20260927 18:39:04.366186 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.327252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:43979 (local address 127.2.95.67:57965)
0927 18:39:04.333574 (+ 6322us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:04.333599 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:04.333989 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:04.344376 (+ 10387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:04.344382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:04.344968 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:04.345238 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.345252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.347131 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.347138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.348729 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.348745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.349958 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.349964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.350305 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.351201 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:04.351237 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:04.355380 (+ 4143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:04.359994 (+ 4614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:04.360006 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:04.360029 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:04.361094 (+ 1065us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:04.361850 (+ 756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:04.361855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:04.361860 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:04.362194 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:04.363077 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:04.363087 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:04.363624 (+ 537us) client_negotiation.cc:769] Sending connection context
0927 18:39:04.363960 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 18:39:04.364523 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6151,"thread_start_us":6414,"threads_started":1}
I20260927 18:39:04.367532 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.327351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:57965 (local address 127.2.95.125:43979)
0927 18:39:04.343963 (+ 16612us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.343970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.344004 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:39:04.344071 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.344077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.344168 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.345396 (+ 1228us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:04.345561 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.346931 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.348963 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.349647 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.350826 (+ 1179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.355675 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:04.355682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:04.355690 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:04.355988 (+ 298us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:04.359771 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.361309 (+ 1538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.361317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.361322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.361544 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.362421 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.362427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.362432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.362850 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:04.362989 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.366821 (+ 3832us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.367110 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16464,"thread_start_us":16365,"threads_started":1}
I20260927 18:39:04.369943 9882 heartbeater.cc:347] Connected to a master server at 127.2.95.125:43979
I20260927 18:39:04.370290 9882 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:04.371434 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.331471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:39795 (local address 127.2.95.67:37529)
0927 18:39:04.338802 (+ 7331us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:04.338824 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:04.339007 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:04.339904 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:04.339920 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:04.340955 (+ 1035us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:04.341995 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.342029 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.345948 (+ 3919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.345955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.348334 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.348361 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.355344 (+ 6983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.355352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.355517 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.356407 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:04.356429 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:04.363286 (+ 6857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:04.367977 (+ 4691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:04.367983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:04.367987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:04.368525 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:04.369300 (+ 775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:04.369304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:04.369307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:04.369444 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:04.370125 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:04.370129 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:04.370286 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 18:39:04.370675 (+ 389us) client_negotiation.cc:240] Negotiation successful
0927 18:39:04.371008 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7192,"thread_start_us":6506,"threads_started":1}
I20260927 18:39:04.372503 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.332816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:37529 (local address 127.2.95.124:39795)
0927 18:39:04.338505 (+ 5689us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.338512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.339122 (+ 610us) server_negotiation.cc:407] Connection header received
0927 18:39:04.339194 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.339199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.339288 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.339378 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:04.342296 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.343467 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.354285 (+ 10818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.354940 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.355062 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.364168 (+ 9106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:04.364175 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:04.364180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:04.364224 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:04.367847 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.369019 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.369025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.369028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.369168 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.369559 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.369564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.369568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.369938 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:04.370484 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.371874 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.372109 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5545,"thread_start_us":5450,"threads_started":1}
I20260927 18:39:04.373359 9886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.310569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.67:47387)
0927 18:39:04.322163 (+ 11594us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:04.322223 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:04.324863 (+ 2640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:04.345274 (+ 20411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:04.345279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:04.345740 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:04.346015 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.346028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.349939 (+ 3911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.349944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.351362 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.351377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.353275 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.353282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.353473 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.355872 (+ 2399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:04.355896 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:04.359278 (+ 3382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:04.364005 (+ 4727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:04.364010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:04.364013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:04.364465 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:04.368598 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:04.368602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:04.368605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:04.368779 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:04.372392 (+ 3613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:04.372397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:04.372539 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 18:39:04.372668 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 18:39:04.372973 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11175,"thread_start_us":15004,"threads_started":1}
I20260927 18:39:04.371788 9882 heartbeater.cc:511] Master 127.2.95.125:43979 requested a full tablet report, sending...
I20260927 18:39:04.374096 9880 heartbeater.cc:347] Connected to a master server at 127.2.95.124:39795
I20260927 18:39:04.374244 9880 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:04.374787 9880 heartbeater.cc:511] Master 127.2.95.124:39795 requested a full tablet report, sending...
I20260927 18:39:04.375567 9216 ts_manager.cc:194] Registered new tserver with Master: db6625f555d64f52b480f453712da79a (127.2.95.67:37053)
I20260927 18:39:04.375914 9413 ts_manager.cc:194] Registered new tserver with Master: db6625f555d64f52b480f453712da79a (127.2.95.67:37053)
I20260927 18:39:04.377465 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.309764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:47387 (local address 127.2.95.126:39071)
0927 18:39:04.344882 (+ 35118us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.344889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.344914 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:39:04.344979 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.344985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.345078 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.345207 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:04.348512 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.349769 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.351538 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.352280 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.352420 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.360053 (+ 7633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:04.360062 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:04.360066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:04.360111 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:04.363886 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.368041 (+ 4155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.368047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.368051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.368486 (+ 435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.369781 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.369786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.369789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.372237 (+ 2448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:04.376748 (+ 4511us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.376888 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.377115 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34975,"thread_start_us":34856,"threads_started":1}
I20260927 18:39:04.378430 9881 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39071
I20260927 18:39:04.378561 9881 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:04.378999 9881 heartbeater.cc:511] Master 127.2.95.126:39071 requested a full tablet report, sending...
I20260927 18:39:04.380024 8618 ts_manager.cc:194] Re-registered known tserver with Master: db6625f555d64f52b480f453712da79a (127.2.95.67:37053)
I20260927 18:39:04.380900 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.349488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.2.95.126:39071)
0927 18:39:04.355441 (+ 5953us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.355447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.355465 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:39:04.356821 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.356827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.356910 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.357020 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:04.363322 (+ 6302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.364528 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.366626 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.367289 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.367419 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.373369 (+ 5950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:04.373376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:04.373381 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:04.373428 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:04.376932 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.377783 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.377789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.377792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.377974 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.378642 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.378647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.378650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.379010 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:04.379253 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.380250 (+ 997us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.380488 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5829,"thread_start_us":6165,"threads_started":1}
I20260927 18:39:04.381198 8618 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:47387
I20260927 18:39:04.386224 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.348940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.2.95.124:39795)
0927 18:39:04.353684 (+ 4744us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.353691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.355833 (+ 2142us) server_negotiation.cc:407] Connection header received
0927 18:39:04.355895 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.355901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.356004 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.356111 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:04.358701 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.360115 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.370548 (+ 10433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.371157 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.371270 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.377136 (+ 5866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:04.377142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:04.377146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:04.377185 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:04.381766 (+ 4581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.382640 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.382646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.382650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.382811 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.384211 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.384217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.384220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.384567 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:04.384818 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.385568 (+ 750us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.385858 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4574,"thread_start_us":5192,"threads_started":1}
I20260927 18:39:04.391426 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.355459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.2.95.125:43979)
0927 18:39:04.356002 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.356008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.371147 (+ 15139us) server_negotiation.cc:407] Connection header received
0927 18:39:04.371229 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.371235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.371386 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.371503 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:04.377089 (+ 5586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.378313 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.380117 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.380775 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.380880 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.385227 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:04.385235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:04.385241 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:04.385289 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:04.388273 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.389198 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.389203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.389205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.389352 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.389849 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.389853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.389855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.390202 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:04.390389 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.390819 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.391066 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":1752,"threads_started":1}
I20260927 18:39:04.386283 8618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:39:04.395291 8618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:39:04.440181 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.415938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39523 (local address 127.0.0.1:51604)
0927 18:39:04.428141 (+ 12203us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:04.428162 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:04.428296 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:04.431487 (+ 3191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:04.431492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:04.431526 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:04.432018 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.432033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.434692 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.434699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.437173 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.437189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.439280 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.439286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.439476 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.439517 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:39:04.439594 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:39:04.439742 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12047,"mutex_wait_us":20,"thread_start_us":18415,"threads_started":1}
I20260927 18:39:04.440806 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.416232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:36525 (local address 127.0.0.1:51154)
0927 18:39:04.431731 (+ 15499us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:04.431755 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:04.431892 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:04.432878 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:04.432882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:04.432906 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:04.433267 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.433278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.435186 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.435192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.438146 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.438163 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.439930 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.439938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.440145 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.440183 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:39:04.440279 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 18:39:04.440430 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15290,"mutex_wait_us":19,"thread_start_us":11649,"threads_started":1}
I20260927 18:39:04.441502 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.416181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.2.95.65:39523)
0927 18:39:04.430748 (+ 14567us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.430758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.430794 (+ 36us) server_negotiation.cc:407] Connection header received
0927 18:39:04.430920 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.430929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.431246 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.431484 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:04.432247 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.434545 (+ 2298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.437466 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.439014 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.440659 (+ 1645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.440883 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.441015 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.441203 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14387,"thread_start_us":14262,"threads_started":1}
I20260927 18:39:04.442900 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.416748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51154 (local address 127.2.95.66:36525)
0927 18:39:04.428786 (+ 12038us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.428794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.432042 (+ 3248us) server_negotiation.cc:407] Connection header received
0927 18:39:04.432158 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.432167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.432449 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.432651 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:04.433498 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.435044 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.438325 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.439694 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.439817 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.440058 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.442479 (+ 2421us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.442637 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11869,"thread_start_us":11738,"threads_started":1}
I20260927 18:39:04.443233 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.428590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.2.95.67:37053)
0927 18:39:04.431227 (+ 2637us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.431235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.434853 (+ 3618us) server_negotiation.cc:407] Connection header received
0927 18:39:04.435102 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.435111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.435409 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.435631 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:04.436331 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.437991 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.440658 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.442165 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.442271 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.442407 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.442783 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.442954 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2479,"thread_start_us":2314,"threads_started":1}
I20260927 18:39:04.444840 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.428350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:37053 (local address 127.0.0.1:54158)
0927 18:39:04.434697 (+ 6347us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:04.434717 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:04.434912 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:04.435627 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:04.435632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:04.435663 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:04.436147 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.436160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.438491 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.438496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.440506 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.440520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.442385 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.442391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.442574 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.442612 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:39:04.442711 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:39:04.442848 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6167,"thread_start_us":2997,"threads_started":1}
I20260927 18:39:04.446444 9674 tablet_service.cc:1467] Processing CreateTablet for tablet f53114636bbf43a2849298ecfeea1287 (DEFAULT_TABLE table=test-table [id=ebc620e67a434742aea4e5bda05eff1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:04.446434 9525 tablet_service.cc:1467] Processing CreateTablet for tablet f53114636bbf43a2849298ecfeea1287 (DEFAULT_TABLE table=test-table [id=ebc620e67a434742aea4e5bda05eff1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:04.447809 9525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53114636bbf43a2849298ecfeea1287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:04.447817 9674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53114636bbf43a2849298ecfeea1287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:04.452991 9815 tablet_service.cc:1467] Processing CreateTablet for tablet f53114636bbf43a2849298ecfeea1287 (DEFAULT_TABLE table=test-table [id=ebc620e67a434742aea4e5bda05eff1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:04.455549 9815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53114636bbf43a2849298ecfeea1287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:04.460868 9914 tablet_bootstrap.cc:492] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36: Bootstrap starting.
I20260927 18:39:04.463735 9915 tablet_bootstrap.cc:492] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660: Bootstrap starting.
I20260927 18:39:04.464811 9914 tablet_bootstrap.cc:654] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:04.466598 9914 log.cc:824] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:04.467803 9915 tablet_bootstrap.cc:654] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:04.469182 9915 log.cc:824] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:04.471455 9918 tablet_bootstrap.cc:492] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a: Bootstrap starting.
I20260927 18:39:04.472652 9915 tablet_bootstrap.cc:492] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660: No bootstrap required, opened a new log
I20260927 18:39:04.472956 9915 ts_tablet_manager.cc:1397] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260927 18:39:04.473276 9914 tablet_bootstrap.cc:492] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36: No bootstrap required, opened a new log
I20260927 18:39:04.473546 9914 ts_tablet_manager.cc:1397] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260927 18:39:04.474604 9918 tablet_bootstrap.cc:654] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:04.475734 9918 log.cc:824] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:04.478559 9918 tablet_bootstrap.cc:492] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a: No bootstrap required, opened a new log
I20260927 18:39:04.478854 9918 ts_tablet_manager.cc:1397] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260927 18:39:04.484817 9915 raft_consensus.cc:348] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } } peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } }
I20260927 18:39:04.484817 9914 raft_consensus.cc:348] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } } peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } }
I20260927 18:39:04.485324 9914 raft_consensus.cc:374] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:04.485324 9915 raft_consensus.cc:374] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:04.485432 9914 raft_consensus.cc:729] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8c6a10b9e94dc0bea3e8e25c261e36, State: Initialized, Role: FOLLOWER
I20260927 18:39:04.485438 9915 raft_consensus.cc:729] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5acedd2a3495c83ed550b02b06660, State: Initialized, Role: FOLLOWER
I20260927 18:39:04.486176 9914 consensus_queue.cc:260] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } } peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } }
I20260927 18:39:04.486174 9915 consensus_queue.cc:260] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } } peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } }
I20260927 18:39:04.487108 9918 raft_consensus.cc:348] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } } peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } }
I20260927 18:39:04.487569 9918 raft_consensus.cc:374] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:04.487669 9918 raft_consensus.cc:729] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6625f555d64f52b480f453712da79a, State: Initialized, Role: FOLLOWER
I20260927 18:39:04.488035 9915 ts_tablet_manager.cc:1428] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 18:39:04.488241 9914 ts_tablet_manager.cc:1428] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 18:39:04.488497 9740 heartbeater.cc:503] Master 127.2.95.126:39071 was elected leader, sending a full tablet report...
I20260927 18:39:04.488791 9918 consensus_queue.cc:260] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } } peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } }
I20260927 18:39:04.497272 9881 heartbeater.cc:503] Master 127.2.95.126:39071 was elected leader, sending a full tablet report...
I20260927 18:39:04.498473 9918 ts_tablet_manager.cc:1428] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
I20260927 18:39:04.504503 9921 raft_consensus.cc:482] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:39:04.504786 9921 raft_consensus.cc:504] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } } peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } }
I20260927 18:39:04.506525 9921 leader_election.cc:288] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba8c6a10b9e94dc0bea3e8e25c261e36 (127.2.95.66:36525), db6625f555d64f52b480f453712da79a (127.2.95.67:37053)
I20260927 18:39:04.513288 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.506845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:35373 (local address 127.2.95.66:36525)
0927 18:39:04.507021 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.507025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.507503 (+ 478us) server_negotiation.cc:407] Connection header received
0927 18:39:04.507757 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.507764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.507839 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.507899 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:04.508495 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.509271 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.511583 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.512367 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.512509 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.512617 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.512890 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.513029 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:39:04.513423 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.507787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:57323 (local address 127.2.95.67:37053)
0927 18:39:04.508107 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.508113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.508132 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:39:04.508255 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.508260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.508315 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.508383 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:04.508818 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.509524 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.511708 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.512560 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.512676 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.512772 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.513072 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.513197 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 18:39:04.513585 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.507501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:37053 (local address 127.2.95.65:57323)
0927 18:39:04.507994 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:04.508012 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:04.508125 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:04.508409 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:04.508413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:04.508445 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:04.508676 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.508683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.509652 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.509655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.511557 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.511570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.512768 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.512774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.512926 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.512957 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:39:04.513043 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 18:39:04.513174 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":252,"threads_started":1}
I20260927 18:39:04.513844 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.506861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:36525 (local address 127.2.95.65:35373)
0927 18:39:04.507402 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:04.507417 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:04.507581 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:04.508062 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:04.508065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:04.508092 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:04.508365 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.508372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.509395 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.509400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.511394 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:04.511409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.512587 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.512594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:04.512744 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.512769 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:39:04.512855 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 18:39:04.512994 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":303,"threads_started":1}
I20260927 18:39:04.514549 9835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53114636bbf43a2849298ecfeea1287" candidate_uuid: "52c5acedd2a3495c83ed550b02b06660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6625f555d64f52b480f453712da79a" is_pre_election: true
I20260927 18:39:04.514675 9694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53114636bbf43a2849298ecfeea1287" candidate_uuid: "52c5acedd2a3495c83ed550b02b06660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" is_pre_election: true
I20260927 18:39:04.515034 9835 raft_consensus.cc:2454] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c5acedd2a3495c83ed550b02b06660 in term 0.
I20260927 18:39:04.515136 9694 raft_consensus.cc:2454] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52c5acedd2a3495c83ed550b02b06660 in term 0.
I20260927 18:39:04.515645 9481 leader_election.cc:302] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c5acedd2a3495c83ed550b02b06660, db6625f555d64f52b480f453712da79a; no voters:
I20260927 18:39:04.516010 9921 raft_consensus.cc:2781] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:39:04.516148 9921 raft_consensus.cc:482] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:39:04.516239 9921 raft_consensus.cc:3037] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:04.518400 9921 raft_consensus.cc:504] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } } peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } }
I20260927 18:39:04.519014 9921 leader_election.cc:288] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [CANDIDATE]: Term 1 election: Requested vote from peers ba8c6a10b9e94dc0bea3e8e25c261e36 (127.2.95.66:36525), db6625f555d64f52b480f453712da79a (127.2.95.67:37053)
I20260927 18:39:04.519572 9835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53114636bbf43a2849298ecfeea1287" candidate_uuid: "52c5acedd2a3495c83ed550b02b06660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6625f555d64f52b480f453712da79a"
I20260927 18:39:04.519654 9694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53114636bbf43a2849298ecfeea1287" candidate_uuid: "52c5acedd2a3495c83ed550b02b06660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36"
I20260927 18:39:04.519771 9835 raft_consensus.cc:3037] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:04.519863 9694 raft_consensus.cc:3037] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:04.522140 9835 raft_consensus.cc:2454] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c5acedd2a3495c83ed550b02b06660 in term 1.
I20260927 18:39:04.522615 9481 leader_election.cc:302] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52c5acedd2a3495c83ed550b02b06660, db6625f555d64f52b480f453712da79a; no voters:
I20260927 18:39:04.522812 9694 raft_consensus.cc:2454] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52c5acedd2a3495c83ed550b02b06660 in term 1.
I20260927 18:39:04.522897 9921 raft_consensus.cc:2781] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:04.523396 9921 raft_consensus.cc:686] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 1 LEADER]: Becoming Leader. State: Replica: 52c5acedd2a3495c83ed550b02b06660, State: Running, Role: LEADER
I20260927 18:39:04.523940 9921 consensus_queue.cc:237] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } } peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } }
I20260927 18:39:04.529022 8618 catalog_manager.cc:5495] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52c5acedd2a3495c83ed550b02b06660 (127.2.95.65). New cstate: current_term: 1 leader_uuid: "52c5acedd2a3495c83ed550b02b06660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } health_report { overall_health: UNKNOWN } } }
I20260927 18:39:04.553319 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.547761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.2.95.65:39523)
0927 18:39:04.547963 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.547969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.547992 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:39:04.548210 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.548214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.548313 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.548451 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:39:04.548801 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.549546 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.551077 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.551510 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.551624 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.552164 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:39:04.552711 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:39:04.552847 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.552966 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.553062 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:39:04.563431 9694 raft_consensus.cc:1261] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 1 FOLLOWER]: Refusing update from remote peer 52c5acedd2a3495c83ed550b02b06660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:04.563453 9835 raft_consensus.cc:1261] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a [term 1 FOLLOWER]: Refusing update from remote peer 52c5acedd2a3495c83ed550b02b06660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:04.564528 9925 consensus_queue.cc:1035] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:04.564944 9921 consensus_queue.cc:1035] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:04.597730 8617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:04.598106 8617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:04.605089 8617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58a7477d2f2741d48e6705163e08facd: Sending DeleteTablet for 3 replicas of tablet f53114636bbf43a2849298ecfeea1287
I20260927 18:39:04.606463 9525 tablet_service.cc:1514] Processing DeleteTablet for tablet f53114636bbf43a2849298ecfeea1287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:04 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:51604
I20260927 18:39:04.606732 9674 tablet_service.cc:1514] Processing DeleteTablet for tablet f53114636bbf43a2849298ecfeea1287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:04 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:51154
I20260927 18:39:04.607206 9815 tablet_service.cc:1514] Processing DeleteTablet for tablet f53114636bbf43a2849298ecfeea1287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:04 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:54158
I20260927 18:39:04.618373 9938 tablet_replica.cc:330] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660: stopping tablet replica
I20260927 18:39:04.618968 9938 raft_consensus.cc:2229] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:39:04.619774 9938 raft_consensus.cc:2258] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:04.622191 9939 tablet_replica.cc:330] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36: stopping tablet replica
I20260927 18:39:04.622704 9939 raft_consensus.cc:2229] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:04.623265 9939 raft_consensus.cc:2258] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:04.624958 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 8586
I20260927 18:39:04.626075 9938 ts_tablet_manager.cc:1905] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:04.626153 8648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:04.634541 9939 ts_tablet_manager.cc:1905] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:04.644022 9938 ts_tablet_manager.cc:1918] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:04.644250 9938 log.cc:1196] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/f53114636bbf43a2849298ecfeea1287
I20260927 18:39:04.644896 9940 tablet_replica.cc:330] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a: stopping tablet replica
I20260927 18:39:04.644935 9938 ts_tablet_manager.cc:1939] T f53114636bbf43a2849298ecfeea1287 P 52c5acedd2a3495c83ed550b02b06660: Deleting consensus metadata
I20260927 18:39:04.645737 9940 raft_consensus.cc:2229] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:04.646324 9940 raft_consensus.cc:2258] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:04.649293 9939 ts_tablet_manager.cc:1918] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:04.649366 9940 ts_tablet_manager.cc:1905] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:04.649475 9939 log.cc:1196] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/f53114636bbf43a2849298ecfeea1287
I20260927 18:39:04.650388 9939 ts_tablet_manager.cc:1939] T f53114636bbf43a2849298ecfeea1287 P ba8c6a10b9e94dc0bea3e8e25c261e36: Deleting consensus metadata
I20260927 18:39:04.656965 9940 ts_tablet_manager.cc:1918] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:04.657256 9940 log.cc:1196] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/f53114636bbf43a2849298ecfeea1287
I20260927 18:39:04.657920 9940 ts_tablet_manager.cc:1939] T f53114636bbf43a2849298ecfeea1287 P db6625f555d64f52b480f453712da79a: Deleting consensus metadata
I20260927 18:39:04.659466 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.610674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.2.95.125:43979)
0927 18:39:04.610876 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.610883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.635055 (+ 24172us) server_negotiation.cc:407] Connection header received
0927 18:39:04.635846 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.635854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.635937 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.636008 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:04.641500 (+ 5492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.642651 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.646279 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.646982 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.647101 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.652100 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:04.652109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:04.652114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:04.652163 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:04.655815 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.656955 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.656962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.656965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.657125 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.657599 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.657605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.657608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.658004 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:04.658244 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.658872 (+ 628us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.659122 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:39:04.677958 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.623975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59180 (local address 127.2.95.124:39795)
0927 18:39:04.624314 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.624321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.660513 (+ 36192us) server_negotiation.cc:407] Connection header received
0927 18:39:04.660787 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.660794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.660879 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.660946 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:04.663022 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.664122 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.666797 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.667418 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.667568 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.671179 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:04.671187 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:04.671192 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:04.671243 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:04.674803 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.675725 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.675731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.675735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.675894 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.676381 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.676387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.676391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.676752 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:04.676992 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.677318 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.677557 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260927 18:39:04.784633 8604 catalog_manager.cc:4841] TS 52c5acedd2a3495c83ed550b02b06660 (127.2.95.65:39523): tablet f53114636bbf43a2849298ecfeea1287 (table test-table [id=ebc620e67a434742aea4e5bda05eff1e]) successfully deleted
I20260927 18:39:04.784790 8605 catalog_manager.cc:4841] TS db6625f555d64f52b480f453712da79a (127.2.95.67:37053): tablet f53114636bbf43a2849298ecfeea1287 (table test-table [id=ebc620e67a434742aea4e5bda05eff1e]) successfully deleted
I20260927 18:39:04.785036 8605 catalog_manager.cc:4841] TS ba8c6a10b9e94dc0bea3e8e25c261e36 (127.2.95.66:36525): tablet f53114636bbf43a2849298ecfeea1287 (table test-table [id=ebc620e67a434742aea4e5bda05eff1e]) successfully deleted
I20260927 18:39:04.785539 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 8586
I20260927 18:39:04.827875 9894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.808337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.2.95.124:39795)
0927 18:39:04.808529 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.808537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.809497 (+ 960us) server_negotiation.cc:407] Connection header received
0927 18:39:04.809801 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.809809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.809940 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.810034 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:04.812698 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.814008 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.816379 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.817143 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.817260 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.821713 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:04.821721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:04.821727 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:04.821775 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:04.824754 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.825621 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.825627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.825631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.825833 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.826208 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.826213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.826216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.826595 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:04.826832 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.827256 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.827492 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:39:04.833853 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:04.809498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.2.95.125:43979)
0927 18:39:04.809711 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:04.809717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:04.818881 (+ 9164us) server_negotiation.cc:407] Connection header received
0927 18:39:04.819152 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:04.819158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:04.819261 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:04.819338 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:04.821660 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.822615 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.824508 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:04.825082 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:04.825196 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:04.828464 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:04.828471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:04.828475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:04.828512 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:04.831220 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.831899 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.831903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.831906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.832039 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:04.832393 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:04.832397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:04.832399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:04.832683 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:04.832907 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:04.833244 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 18:39:04.833467 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:39:05.592229 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:05.591454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.65:34231)
0927 18:39:05.591870 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:05.592067 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":218,"threads_started":1}
W20260927 18:39:05.592631 9593 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:39071 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
I20260927 18:39:05.597564 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:05.596655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.66:47395)
0927 18:39:05.597107 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:05.597265 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":230,"threads_started":1}
W20260927 18:39:05.598032 9740 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:39071 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
I20260927 18:39:05.610208 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:05.609287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.67:43249)
0927 18:39:05.609769 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:05.609915 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":210,"threads_started":1}
W20260927 18:39:05.610590 9881 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:39071 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
I20260927 18:39:06.105746 9953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58a7477d2f2741d48e6705163e08facd)
I20260927 18:39:06.106127 9953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } }
I20260927 18:39:06.107306 9954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58a7477d2f2741d48e6705163e08facd)
I20260927 18:39:06.107474 9953 leader_election.cc:288] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58a7477d2f2741d48e6705163e08facd (127.2.95.126:39071), 9c9618bf0f8648ed987531e4f6839082 (127.2.95.125:43979)
I20260927 18:39:06.107511 9954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } }
I20260927 18:39:06.108670 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.107697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:59100)
0927 18:39:06.108183 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.108348 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":253,"threads_started":1}
I20260927 18:39:06.109738 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.108537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:59116)
0927 18:39:06.109268 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.109457 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":459,"threads_started":1}
W20260927 18:39:06.110101 9201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
W20260927 18:39:06.110111 9398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
I20260927 18:39:06.114693 9954 leader_election.cc:288] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58a7477d2f2741d48e6705163e08facd (127.2.95.126:39071), 346e593c3daa4b208ada0a83c81199e5 (127.2.95.124:39795)
I20260927 18:39:06.121752 9957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.120745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:59132)
0927 18:39:06.121311 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.121481 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414}
W20260927 18:39:06.121973 9201 leader_election.cc:334] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58a7477d2f2741d48e6705163e08facd (127.2.95.126:39071): Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
I20260927 18:39:06.122334 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.116692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:59120)
0927 18:39:06.121500 (+ 4808us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.121725 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4590,"thread_start_us":4415,"threads_started":1}
W20260927 18:39:06.122633 9398 leader_election.cc:334] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58a7477d2f2741d48e6705163e08facd (127.2.95.126:39071): Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
I20260927 18:39:06.125346 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.109267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:43979 (local address 127.0.0.1:52036)
0927 18:39:06.109546 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.109566 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:06.109737 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:06.117132 (+ 7395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:06.117139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:06.117180 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:06.117463 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.117477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.121402 (+ 3925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.121409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.123589 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.123602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.124800 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.124804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.124921 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.124950 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:39:06.125039 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:39:06.125151 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260927 18:39:06.125692 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.108459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52036 (local address 127.2.95.125:43979)
0927 18:39:06.115901 (+ 7442us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:06.115908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:06.115936 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:39:06.116018 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:06.116024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:06.116107 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:06.117723 (+ 1616us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:06.117957 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.119381 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.123749 (+ 4368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.124620 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.124778 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.124879 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:06.125079 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 18:39:06.125223 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7302,"thread_start_us":7081,"threads_started":1}
I20260927 18:39:06.126371 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "346e593c3daa4b208ada0a83c81199e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9c9618bf0f8648ed987531e4f6839082" is_pre_election: true
I20260927 18:39:06.126737 9226 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346e593c3daa4b208ada0a83c81199e5 in term 1.
I20260927 18:39:06.127398 9398 leader_election.cc:302] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346e593c3daa4b208ada0a83c81199e5, 9c9618bf0f8648ed987531e4f6839082; no voters: 58a7477d2f2741d48e6705163e08facd
I20260927 18:39:06.127970 9953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 18:39:06.128116 9953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58a7477d2f2741d48e6705163e08facd)
I20260927 18:39:06.128216 9953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:39:06.128752 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.108554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:39795 (local address 127.0.0.1:48014)
0927 18:39:06.115230 (+ 6676us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.115255 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:06.115466 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:06.123279 (+ 7813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:06.123284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:06.123313 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:06.123546 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.123555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.124571 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.124581 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.126427 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.126441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.127889 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.127896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.128081 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.128121 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:39:06.128250 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 18:39:06.128444 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6554,"mutex_wait_us":24,"thread_start_us":13639,"threads_started":1}
I20260927 18:39:06.129660 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.108899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48014 (local address 127.2.95.124:39795)
0927 18:39:06.122915 (+ 14016us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:06.122921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:06.122949 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:39:06.123016 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:06.123022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:06.123107 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:06.123212 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:06.123702 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.124434 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.126585 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.127693 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.127817 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.127994 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:06.129077 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0927 18:39:06.129229 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13771,"thread_start_us":13634,"threads_started":1}
I20260927 18:39:06.130517 9423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c9618bf0f8648ed987531e4f6839082" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "346e593c3daa4b208ada0a83c81199e5" is_pre_election: true
I20260927 18:39:06.131644 9953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } }
I20260927 18:39:06.132200 9423 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c9618bf0f8648ed987531e4f6839082 in current term 2: Already voted for candidate 346e593c3daa4b208ada0a83c81199e5 in this term.
I20260927 18:39:06.132413 9953 leader_election.cc:288] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [CANDIDATE]: Term 2 election: Requested vote from peers 58a7477d2f2741d48e6705163e08facd (127.2.95.126:39071), 9c9618bf0f8648ed987531e4f6839082 (127.2.95.125:43979)
I20260927 18:39:06.133134 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.132705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:59158)
0927 18:39:06.132874 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.132977 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 18:39:06.133239 9200 leader_election.cc:302] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c9618bf0f8648ed987531e4f6839082; no voters: 346e593c3daa4b208ada0a83c81199e5, 58a7477d2f2741d48e6705163e08facd
I20260927 18:39:06.133284 9226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "346e593c3daa4b208ada0a83c81199e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9c9618bf0f8648ed987531e4f6839082"
I20260927 18:39:06.133508 9226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:39:06.134366 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.133979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:59168)
0927 18:39:06.134138 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.134231 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 18:39:06.134586 9398 leader_election.cc:334] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 58a7477d2f2741d48e6705163e08facd (127.2.95.126:39071): Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
I20260927 18:39:06.135978 9226 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346e593c3daa4b208ada0a83c81199e5 in term 2.
I20260927 18:39:06.136147 9954 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:39:06.136410 9398 leader_election.cc:302] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346e593c3daa4b208ada0a83c81199e5, 9c9618bf0f8648ed987531e4f6839082; no voters: 58a7477d2f2741d48e6705163e08facd
I20260927 18:39:06.136695 9953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:39:06.137147 9953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [term 2 LEADER]: Becoming Leader. State: Replica: 346e593c3daa4b208ada0a83c81199e5, State: Running, Role: LEADER
I20260927 18:39:06.137581 9953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } }
I20260927 18:39:06.139614 9963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 346e593c3daa4b208ada0a83c81199e5. Latest consensus state: current_term: 2 leader_uuid: "346e593c3daa4b208ada0a83c81199e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } } }
I20260927 18:39:06.140055 9963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:06.140794 9965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:39:06.144635 9965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:39:06.145531 9965 catalog_manager.cc:1259] Loaded cluster ID: dac25e4c532440a4ba8372b8b7af6383
I20260927 18:39:06.145609 9965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:39:06.146999 9965 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:39:06.147866 9965 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: Loaded TSK: 0
I20260927 18:39:06.148727 9965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:39:06.188835 9413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:39:06.196035 9413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:39:06.200688 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.200217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:59178)
0927 18:39:06.200390 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.200480 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 18:39:06.201095 9226 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [term 2 FOLLOWER]: Refusing update from remote peer 346e593c3daa4b208ada0a83c81199e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 18:39:06.201896 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.201478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:59188)
0927 18:39:06.201631 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.201765 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 18:39:06.202092 9963 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 18:39:06.202365 9398 consensus_peers.cc:487] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 -> Peer 58a7477d2f2741d48e6705163e08facd (127.2.95.126:39071): Couldn't send request to peer 58a7477d2f2741d48e6705163e08facd. Status: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:39:06.204910 9954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 346e593c3daa4b208ada0a83c81199e5. Latest consensus state: current_term: 2 leader_uuid: "346e593c3daa4b208ada0a83c81199e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } } }
I20260927 18:39:06.205204 9954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:39:06.206861 9963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "346e593c3daa4b208ada0a83c81199e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } } }
I20260927 18:39:06.207264 9963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5 [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:06.207804 9954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "346e593c3daa4b208ada0a83c81199e5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "58a7477d2f2741d48e6705163e08facd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39071 } } peers { permanent_uuid: "9c9618bf0f8648ed987531e4f6839082" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 43979 } attrs { promote: false } } peers { permanent_uuid: "346e593c3daa4b208ada0a83c81199e5" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 39795 } attrs { promote: false } } }
I20260927 18:39:06.208091 9954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9618bf0f8648ed987531e4f6839082 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:39:06.231797 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.216012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:36525 (local address 127.0.0.1:50426)
0927 18:39:06.216176 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.216195 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:06.216321 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:06.226702 (+ 10381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:06.226707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:06.226735 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:06.226893 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.226902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.227871 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.227877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.229417 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.229429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.230985 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.230990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.231177 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.231215 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:39:06.231316 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:39:06.231461 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":1}
I20260927 18:39:06.231833 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.215972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.2.95.66:36525)
0927 18:39:06.226399 (+ 10427us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:06.226405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:06.226425 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:39:06.226493 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:06.226498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:06.226567 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:06.226659 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:06.227033 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.227734 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.229609 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.230795 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.230956 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.231104 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:06.231371 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 18:39:06.231528 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10312,"thread_start_us":10467,"threads_started":1}
I20260927 18:39:06.232713 9674 tablet_service.cc:1467] Processing CreateTablet for tablet 578f525ad0c449cd9c376f4a945d6fba (DEFAULT_TABLE table=test-table [id=dda3e6545d0b41d4ae20459938d4f422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:06.233180 9674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578f525ad0c449cd9c376f4a945d6fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:06.234592 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.216333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:37053 (local address 127.0.0.1:56280)
0927 18:39:06.216930 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.216950 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:06.217067 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:06.228742 (+ 11675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:06.228747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:06.228778 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:06.229159 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.229175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.230521 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.230527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.232423 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.232437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.233823 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.233835 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.234059 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.234094 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:39:06.234180 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 18:39:06.234325 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477}
I20260927 18:39:06.235512 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.216683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.2.95.67:37053)
0927 18:39:06.228237 (+ 11554us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:06.228244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:06.228275 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:39:06.228355 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:06.228361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:06.228485 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:06.229318 (+ 833us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:06.229515 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.230385 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.232562 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.233557 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.234908 (+ 1351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.235049 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:06.235103 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 18:39:06.235257 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11407,"thread_start_us":11894,"threads_started":1}
I20260927 18:39:06.236653 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.216535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39523 (local address 127.0.0.1:41816)
0927 18:39:06.226209 (+ 9674us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.226231 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:06.226367 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:06.230368 (+ 4001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:06.230373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:06.230400 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:06.230747 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.230757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.232149 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.232154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.234278 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.234291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.235995 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.236008 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.236186 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.236211 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:39:06.236290 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:39:06.236392 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9525,"thread_start_us":9331,"threads_started":1}
I20260927 18:39:06.236773 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.216636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.2.95.65:39523)
0927 18:39:06.230045 (+ 13409us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:06.230051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:06.230071 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:39:06.230134 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:06.230139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:06.230222 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:06.230326 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:06.230905 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.231872 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.234467 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.235706 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.235833 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.235936 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:06.236336 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 18:39:06.236495 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13279,"thread_start_us":13186,"threads_started":1}
I20260927 18:39:06.237550 9525 tablet_service.cc:1467] Processing CreateTablet for tablet 578f525ad0c449cd9c376f4a945d6fba (DEFAULT_TABLE table=test-table [id=dda3e6545d0b41d4ae20459938d4f422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:06.237872 9815 tablet_service.cc:1467] Processing CreateTablet for tablet 578f525ad0c449cd9c376f4a945d6fba (DEFAULT_TABLE table=test-table [id=dda3e6545d0b41d4ae20459938d4f422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:06.238162 9525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578f525ad0c449cd9c376f4a945d6fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:06.238248 9815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578f525ad0c449cd9c376f4a945d6fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:06.239677 9976 tablet_bootstrap.cc:492] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36: Bootstrap starting.
I20260927 18:39:06.241156 9976 tablet_bootstrap.cc:654] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:06.243139 9976 tablet_bootstrap.cc:492] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36: No bootstrap required, opened a new log
I20260927 18:39:06.243273 9976 ts_tablet_manager.cc:1397] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 18:39:06.243889 9976 raft_consensus.cc:348] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } } peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } }
I20260927 18:39:06.244095 9976 raft_consensus.cc:374] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:06.244184 9976 raft_consensus.cc:729] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8c6a10b9e94dc0bea3e8e25c261e36, State: Initialized, Role: FOLLOWER
I20260927 18:39:06.244398 9976 consensus_queue.cc:260] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } } peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } }
I20260927 18:39:06.244998 9978 tablet_bootstrap.cc:492] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660: Bootstrap starting.
I20260927 18:39:06.246227 9980 tablet_bootstrap.cc:492] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a: Bootstrap starting.
I20260927 18:39:06.247004 9978 tablet_bootstrap.cc:654] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:06.247718 9980 tablet_bootstrap.cc:654] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:06.248607 9976 ts_tablet_manager.cc:1428] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 18:39:06.249454 9978 tablet_bootstrap.cc:492] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660: No bootstrap required, opened a new log
I20260927 18:39:06.249612 9978 ts_tablet_manager.cc:1397] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 18:39:06.251036 9978 raft_consensus.cc:348] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } } peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } }
I20260927 18:39:06.251345 9978 raft_consensus.cc:374] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:06.251446 9978 raft_consensus.cc:729] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c5acedd2a3495c83ed550b02b06660, State: Initialized, Role: FOLLOWER
I20260927 18:39:06.251665 9978 consensus_queue.cc:260] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } } peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } }
I20260927 18:39:06.254760 9739 heartbeater.cc:503] Master 127.2.95.124:39795 was elected leader, sending a full tablet report...
I20260927 18:39:06.257061 9978 ts_tablet_manager.cc:1428] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260927 18:39:06.261047 9592 heartbeater.cc:503] Master 127.2.95.124:39795 was elected leader, sending a full tablet report...
I20260927 18:39:06.268328 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.249223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.66:42961)
0927 18:39:06.267867 (+ 18644us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.268028 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18504,"thread_start_us":18348,"threads_started":1}
I20260927 18:39:06.270138 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.269380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.66:38683)
0927 18:39:06.269923 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.270005 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428}
I20260927 18:39:06.271693 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.258281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.65:48275)
0927 18:39:06.269622 (+ 11341us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.269788 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11174,"thread_start_us":11016,"threads_started":1}
I20260927 18:39:06.272898 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.272500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.65:44997)
0927 18:39:06.272665 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.272750 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 18:39:06.273277 9980 tablet_bootstrap.cc:492] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a: No bootstrap required, opened a new log
I20260927 18:39:06.273450 9980 ts_tablet_manager.cc:1397] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a: Time spent bootstrapping tablet: real 0.027s user 0.026s sys 0.000s
I20260927 18:39:06.274179 9980 raft_consensus.cc:348] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } } peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } }
I20260927 18:39:06.274381 9980 raft_consensus.cc:374] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:06.274471 9980 raft_consensus.cc:729] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6625f555d64f52b480f453712da79a, State: Initialized, Role: FOLLOWER
I20260927 18:39:06.274711 9980 consensus_queue.cc:260] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } } peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } }
I20260927 18:39:06.275458 9980 ts_tablet_manager.cc:1428] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:39:06.276727 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.275835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.67:37045)
0927 18:39:06.276352 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.276486 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":266,"threads_started":1}
I20260927 18:39:06.277558 9880 heartbeater.cc:503] Master 127.2.95.124:39795 was elected leader, sending a full tablet report...
I20260927 18:39:06.278400 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.278059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.67:57377)
0927 18:39:06.278202 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.278285 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 18:39:06.308857 9979 raft_consensus.cc:482] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:39:06.309190 9979 raft_consensus.cc:504] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } } peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } }
I20260927 18:39:06.310911 9979 leader_election.cc:288] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db6625f555d64f52b480f453712da79a (127.2.95.67:37053), 52c5acedd2a3495c83ed550b02b06660 (127.2.95.65:39523)
I20260927 18:39:06.318473 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.311266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:37053 (local address 127.2.95.66:44791)
0927 18:39:06.311526 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.311547 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:06.311853 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:06.312535 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:06.312540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:06.312563 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:06.312801 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.312812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.314199 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.314205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.316394 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.316411 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.317814 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.317819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.318025 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.318052 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 18:39:06.318128 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 18:39:06.318239 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260927 18:39:06.318573 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.311333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:44791 (local address 127.2.95.67:37053)
0927 18:39:06.311566 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:06.311573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:06.311651 (+ 78us) server_negotiation.cc:407] Connection header received
0927 18:39:06.312096 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:06.312102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:06.312183 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:06.312266 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:06.312976 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.314078 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.316593 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.317590 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.317751 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.317860 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:06.318182 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 18:39:06.318328 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 18:39:06.319247 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.311316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:52561 (local address 127.2.95.65:39523)
0927 18:39:06.311600 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:06.311605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:06.312431 (+ 826us) server_negotiation.cc:407] Connection header received
0927 18:39:06.312626 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:06.312632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:06.312715 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:06.312802 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:06.313365 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.314490 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.317130 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.318347 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.318475 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.318581 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:06.318932 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 18:39:06.319057 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260927 18:39:06.319312 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.311743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39523 (local address 127.2.95.66:52561)
0927 18:39:06.312333 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.312351 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:06.312478 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:06.312837 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:06.312841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:06.312873 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:06.313182 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.313192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.314723 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.314758 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.316936 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:06.316950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.318575 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.318581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:06.318770 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.318801 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:39:06.318884 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 18:39:06.319015 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":315,"threads_started":1}
I20260927 18:39:06.319286 9835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578f525ad0c449cd9c376f4a945d6fba" candidate_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6625f555d64f52b480f453712da79a" is_pre_election: true
I20260927 18:39:06.319563 9835 raft_consensus.cc:2454] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba8c6a10b9e94dc0bea3e8e25c261e36 in term 0.
I20260927 18:39:06.320133 9630 leader_election.cc:302] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba8c6a10b9e94dc0bea3e8e25c261e36, db6625f555d64f52b480f453712da79a; no voters:
I20260927 18:39:06.320348 9547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578f525ad0c449cd9c376f4a945d6fba" candidate_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5acedd2a3495c83ed550b02b06660" is_pre_election: true
I20260927 18:39:06.320600 9979 raft_consensus.cc:2781] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:39:06.320708 9979 raft_consensus.cc:482] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:39:06.320801 9979 raft_consensus.cc:3037] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:06.320973 9547 raft_consensus.cc:2454] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba8c6a10b9e94dc0bea3e8e25c261e36 in term 0.
I20260927 18:39:06.323261 9979 raft_consensus.cc:504] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } } peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } }
I20260927 18:39:06.323902 9979 leader_election.cc:288] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [CANDIDATE]: Term 1 election: Requested vote from peers db6625f555d64f52b480f453712da79a (127.2.95.67:37053), 52c5acedd2a3495c83ed550b02b06660 (127.2.95.65:39523)
I20260927 18:39:06.324494 9835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578f525ad0c449cd9c376f4a945d6fba" candidate_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db6625f555d64f52b480f453712da79a"
I20260927 18:39:06.324494 9547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578f525ad0c449cd9c376f4a945d6fba" candidate_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c5acedd2a3495c83ed550b02b06660"
I20260927 18:39:06.324703 9835 raft_consensus.cc:3037] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:06.324703 9547 raft_consensus.cc:3037] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:06.327658 9835 raft_consensus.cc:2454] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba8c6a10b9e94dc0bea3e8e25c261e36 in term 1.
I20260927 18:39:06.328239 9630 leader_election.cc:302] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba8c6a10b9e94dc0bea3e8e25c261e36, db6625f555d64f52b480f453712da79a; no voters:
I20260927 18:39:06.328471 9547 raft_consensus.cc:2454] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba8c6a10b9e94dc0bea3e8e25c261e36 in term 1.
I20260927 18:39:06.328603 9979 raft_consensus.cc:2781] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:06.329088 9979 raft_consensus.cc:686] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 1 LEADER]: Becoming Leader. State: Replica: ba8c6a10b9e94dc0bea3e8e25c261e36, State: Running, Role: LEADER
I20260927 18:39:06.329469 9979 consensus_queue.cc:237] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } } peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } }
I20260927 18:39:06.330405 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.329918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.66:56265)
0927 18:39:06.330108 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.330248 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 18:39:06.333241 9413 catalog_manager.cc:5495] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba8c6a10b9e94dc0bea3e8e25c261e36 (127.2.95.66). New cstate: current_term: 1 leader_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8c6a10b9e94dc0bea3e8e25c261e36" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 } health_report { overall_health: UNKNOWN } } }
W20260927 18:39:06.337409 9883 tablet.cc:2369] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:39:06.346890 9595 tablet.cc:2369] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:39:06.357254 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.351640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.2.95.66:36525)
0927 18:39:06.351854 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:06.351859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:06.351878 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:39:06.352120 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:06.352124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:06.352205 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:06.352328 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:39:06.352674 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.353440 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.354969 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.355417 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.355547 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.355992 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:39:06.356645 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:39:06.356759 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:06.356922 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 18:39:06.356997 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:39:06.362214 9835 raft_consensus.cc:1261] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a [term 1 FOLLOWER]: Refusing update from remote peer ba8c6a10b9e94dc0bea3e8e25c261e36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:06.362498 9547 raft_consensus.cc:1261] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660 [term 1 FOLLOWER]: Refusing update from remote peer ba8c6a10b9e94dc0bea3e8e25c261e36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:06.363137 9989 consensus_queue.cc:1035] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6625f555d64f52b480f453712da79a" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 37053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:06.363570 9979 consensus_queue.cc:1035] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c5acedd2a3495c83ed550b02b06660" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:06.365785 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.365275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.67:53703)
0927 18:39:06.365453 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.365568 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 18:39:06.366477 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.365932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.65:51403)
0927 18:39:06.366169 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.366282 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260927 18:39:06.373893 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.373301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.67:50125)
0927 18:39:06.373551 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.373658 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 18:39:06.378656 9994 mvcc.cc:205] Tried to move back new op lower bound from 7334028682690371584 to 7334028682570260480. Current Snapshot: MvccSnapshot[applied={T|T < 7334028682690371584}]
I20260927 18:39:06.379802 9993 mvcc.cc:205] Tried to move back new op lower bound from 7334028682690371584 to 7334028682570260480. Current Snapshot: MvccSnapshot[applied={T|T < 7334028682690371584}]
I20260927 18:39:06.380374 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.379749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.66:40153)
0927 18:39:06.380110 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.380207 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255}
I20260927 18:39:06.381425 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.380964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.2.95.65:57751)
0927 18:39:06.381204 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.381288 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260927 18:39:06.390594 9412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:06.390842 9412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:06.397552 9412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 346e593c3daa4b208ada0a83c81199e5: Sending DeleteTablet for 3 replicas of tablet 578f525ad0c449cd9c376f4a945d6fba
I20260927 18:39:06.398917 9815 tablet_service.cc:1514] Processing DeleteTablet for tablet 578f525ad0c449cd9c376f4a945d6fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:06 UTC) from {username='oryx', principal='oryx/127.2.95.124@KRBTEST.COM'} at 127.0.0.1:56280
I20260927 18:39:06.398960 9674 tablet_service.cc:1514] Processing DeleteTablet for tablet 578f525ad0c449cd9c376f4a945d6fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:06 UTC) from {username='oryx', principal='oryx/127.2.95.124@KRBTEST.COM'} at 127.0.0.1:50426
I20260927 18:39:06.398990 9525 tablet_service.cc:1514] Processing DeleteTablet for tablet 578f525ad0c449cd9c376f4a945d6fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:06 UTC) from {username='oryx', principal='oryx/127.2.95.124@KRBTEST.COM'} at 127.0.0.1:41816
I20260927 18:39:06.400121 10004 tablet_replica.cc:330] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660: stopping tablet replica
I20260927 18:39:06.400328 10004 raft_consensus.cc:2229] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:06.400619 10004 raft_consensus.cc:2258] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:06.401695 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 9385
I20260927 18:39:06.405936 10004 ts_tablet_manager.cc:1905] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:06.413571 10004 ts_tablet_manager.cc:1918] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:06.413851 10004 log.cc:1196] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/578f525ad0c449cd9c376f4a945d6fba
I20260927 18:39:06.414306 10002 tablet_replica.cc:330] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36: stopping tablet replica
I20260927 18:39:06.414465 10004 ts_tablet_manager.cc:1939] T 578f525ad0c449cd9c376f4a945d6fba P 52c5acedd2a3495c83ed550b02b06660: Deleting consensus metadata
I20260927 18:39:06.414664 10002 raft_consensus.cc:2229] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:39:06.415364 10002 raft_consensus.cc:2258] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:06.415892 9399 catalog_manager.cc:4841] TS 52c5acedd2a3495c83ed550b02b06660 (127.2.95.65:39523): tablet 578f525ad0c449cd9c376f4a945d6fba (table test-table [id=dda3e6545d0b41d4ae20459938d4f422]) successfully deleted
I20260927 18:39:06.416170 10003 tablet_replica.cc:330] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a: stopping tablet replica
I20260927 18:39:06.416404 10003 raft_consensus.cc:2229] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:06.416719 10002 ts_tablet_manager.cc:1905] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:06.416728 10003 raft_consensus.cc:2258] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:06.418002 10003 ts_tablet_manager.cc:1905] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:06.425081 10003 ts_tablet_manager.cc:1918] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:06.425247 10003 log.cc:1196] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/578f525ad0c449cd9c376f4a945d6fba
I20260927 18:39:06.425661 10003 ts_tablet_manager.cc:1939] T 578f525ad0c449cd9c376f4a945d6fba P db6625f555d64f52b480f453712da79a: Deleting consensus metadata
I20260927 18:39:06.427098 9400 catalog_manager.cc:4841] TS db6625f555d64f52b480f453712da79a (127.2.95.67:37053): tablet 578f525ad0c449cd9c376f4a945d6fba (table test-table [id=dda3e6545d0b41d4ae20459938d4f422]) successfully deleted
I20260927 18:39:06.430277 10002 ts_tablet_manager.cc:1918] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:06.430424 10002 log.cc:1196] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/578f525ad0c449cd9c376f4a945d6fba
I20260927 18:39:06.430927 10002 ts_tablet_manager.cc:1939] T 578f525ad0c449cd9c376f4a945d6fba P ba8c6a10b9e94dc0bea3e8e25c261e36: Deleting consensus metadata
I20260927 18:39:06.433907 9400 catalog_manager.cc:4841] TS ba8c6a10b9e94dc0bea3e8e25c261e36 (127.2.95.66:36525): tablet 578f525ad0c449cd9c376f4a945d6fba (table test-table [id=dda3e6545d0b41d4ae20459938d4f422]) successfully deleted
I20260927 18:39:06.434864 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.403315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.2.95.65:39523)
0927 18:39:06.403540 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:06.403546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:06.410883 (+ 7337us) server_negotiation.cc:407] Connection header received
0927 18:39:06.411904 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:06.411910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:06.412023 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:06.412100 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:06.417129 (+ 5029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.418182 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.420118 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.420762 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.420901 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.424785 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:06.424807 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:06.424828 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:06.424877 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:06.429367 (+ 4490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:06.430636 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:06.430647 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:06.430651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:06.430830 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:06.431301 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:06.431306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:06.431309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:06.431814 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:06.433940 (+ 2126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:06.434228 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 18:39:06.434568 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:39:06.435967 9587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:06.556787 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 9385
I20260927 18:39:06.585166 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 9604
I20260927 18:39:06.602945 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.586098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.2.95.66:36525)
0927 18:39:06.586261 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:06.586266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:06.586285 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:39:06.586525 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:06.586530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:06.586631 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:06.586726 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:06.588428 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.589309 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.591250 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.591720 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.591824 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.595144 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:06.595162 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:06.595180 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:06.595238 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:06.599197 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:06.600242 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:06.600253 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:06.600257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:06.600524 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:06.601023 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:06.601029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:06.601033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:06.601524 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:06.601914 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:06.602262 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 18:39:06.602530 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:39:06.603731 9734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:06.685905 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.685367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:59190)
0927 18:39:06.685570 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.685727 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 18:39:06.687106 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.686686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39071 (local address 127.0.0.1:59198)
0927 18:39:06.686835 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:06.686926 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 18:39:06.725620 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 9604
I20260927 18:39:06.742563 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 9752
I20260927 18:39:06.760416 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:06.743407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.2.95.67:37053)
0927 18:39:06.743665 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:06.743670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:06.743691 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:39:06.743912 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:06.743917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:06.743997 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:06.744083 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:06.745576 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.746677 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.748468 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:06.749183 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:06.749372 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:06.753416 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:06.753439 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:06.753460 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:06.753512 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:06.757051 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:06.758008 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:06.758016 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:06.758019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:06.758190 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:06.758646 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:06.758651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:06.758653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:06.759178 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:06.759427 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:06.759802 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 18:39:06.760084 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 18:39:06.761034 9875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:06.854426 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 9752
I20260927 18:39:06.870386 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 9079
I20260927 18:39:06.871534 9246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:06.988416 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 9079
I20260927 18:39:07.005237 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 9179
I20260927 18:39:07.006405 9443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:07.131299 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 9179
2026-09-27T18:39:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9512 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:39:07 dist-test-slave-9jll krb5kdc[10013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:39:07 dist-test-slave-9jll krb5kdc[10013](info): set up 2 sockets
Sep 27 18:39:07 dist-test-slave-9jll krb5kdc[10013](info): commencing operation
krb5kdc: starting...
W20260927 18:39:08.166445 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:39:08 dist-test-slave-9jll krb5kdc[10013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:39:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.126.keytab.
Entry for principal oryx/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.126.keytab.
I20260927 18:39:08.389919 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:37211
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.126.keytab
--principal=oryx/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:37211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:08.719529 10026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:08.719841 10026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:08.719973 10026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:08.727586 10026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:39:08.727723 10026 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 18:39:08.727766 10026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:08.727850 10026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:08.727926 10026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:39:08.728000 10026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:39:08.738334 10026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:37211
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.126.keytab
--principal=oryx/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:37211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:08.739665 10026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:08.741052 10026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:08.753500 10031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:08.753500 10032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:08.753854 10034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:08.754324 10026 server_base.cc:1034] running on GCE node
I20260927 18:39:08.755107 10026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:08.756596 10026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:08.757807 10026 hybrid_clock.cc:648] HybridClock initialized: now 1790534348757721 us; error 54 us; skew 500 ppm
I20260927 18:39:08.758131 10026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:08 dist-test-slave-9jll krb5kdc[10013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534348, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:08.761512 10026 init.cc:377] Logged in from keytab as oryx/127.2.95.126@KRBTEST.COM (short username oryx)
I20260927 18:39:08.763206 10026 webserver.cc:460] Webserver started at http://127.2.95.126:45499/ using document root <none> and password file <none>
I20260927 18:39:08.763948 10026 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:08.764055 10026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:08.764338 10026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:08.767400 10026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "0d7754abcb1a40e192502fcd881c54f6"
format_stamp: "Formatted at 2026-09-27 18:39:08 on dist-test-slave-9jll"
server_key: "4ac0885c020d160cb6d9b063ace3b988"
server_key_iv: "a9f3a2a0c1866d2fd1502148d1fdc0ac"
server_key_version: "encryptionkey@0"
I20260927 18:39:08.768178 10026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "0d7754abcb1a40e192502fcd881c54f6"
format_stamp: "Formatted at 2026-09-27 18:39:08 on dist-test-slave-9jll"
server_key: "4ac0885c020d160cb6d9b063ace3b988"
server_key_iv: "a9f3a2a0c1866d2fd1502148d1fdc0ac"
server_key_version: "encryptionkey@0"
I20260927 18:39:08.773488 10026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:39:08.777122 10041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:08.778512 10026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:39:08.778683 10026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "0d7754abcb1a40e192502fcd881c54f6"
format_stamp: "Formatted at 2026-09-27 18:39:08 on dist-test-slave-9jll"
server_key: "4ac0885c020d160cb6d9b063ace3b988"
server_key_iv: "a9f3a2a0c1866d2fd1502148d1fdc0ac"
server_key_version: "encryptionkey@0"
I20260927 18:39:08.778887 10026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:08.799254 10026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:08.803961 10026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:08.804227 10026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:08.825475 10026 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:37211
I20260927 18:39:08.825498 10102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:37211 every 8 connection(s)
I20260927 18:39:08.826928 10026 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:39:08.831156 10103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:08.831745 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 10026
I20260927 18:39:08.831924 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:39:08.832564 2429 external_mini_cluster.cc:1428] Setting key 60eaa27628273c269cf39a4986c993a2
I20260927 18:39:08.846307 10103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6: Bootstrap starting.
Sep 27 18:39:08 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:08.849588 10103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:08.851141 10103 log.cc:824] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:08.854741 10103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6: No bootstrap required, opened a new log
I20260927 18:39:08.858714 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:08.834628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.2.95.126:37211)
0927 18:39:08.835528 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:08.835544 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:08.835597 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:39:08.837645 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:08.837704 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:08.838297 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:08.838791 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:08.840543 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:08.842613 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:08.844370 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:08.845595 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:08.845831 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:08.849887 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:08.849899 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:08.850009 (+ 110us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:08.850113 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:08.854102 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:08.855013 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:08.855020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:08.855028 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:08.855298 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:08.855689 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:08.855693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:08.855695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:08.856295 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:08.856630 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:08.857150 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 18:39:08.857613 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":392,"threads_started":1}
I20260927 18:39:08.863600 10103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7754abcb1a40e192502fcd881c54f6" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37211 } }
I20260927 18:39:08.864008 10103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:08.864099 10103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7754abcb1a40e192502fcd881c54f6, State: Initialized, Role: FOLLOWER
I20260927 18:39:08.864729 10103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7754abcb1a40e192502fcd881c54f6" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37211 } }
I20260927 18:39:08.865013 10103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:39:08.865193 10103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:39:08.865377 10103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:08.868585 10103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7754abcb1a40e192502fcd881c54f6" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37211 } }
I20260927 18:39:08.869145 10103 leader_election.cc:302] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d7754abcb1a40e192502fcd881c54f6; no voters:
I20260927 18:39:08.869781 10103 leader_election.cc:288] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:39:08.869915 10108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:08.870895 10108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [term 1 LEADER]: Becoming Leader. State: Replica: 0d7754abcb1a40e192502fcd881c54f6, State: Running, Role: LEADER
I20260927 18:39:08.871471 10108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7754abcb1a40e192502fcd881c54f6" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37211 } }
I20260927 18:39:08.871945 10103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:39:08.874028 10110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d7754abcb1a40e192502fcd881c54f6. Latest consensus state: current_term: 1 leader_uuid: "0d7754abcb1a40e192502fcd881c54f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7754abcb1a40e192502fcd881c54f6" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37211 } } }
I20260927 18:39:08.874022 10109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d7754abcb1a40e192502fcd881c54f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7754abcb1a40e192502fcd881c54f6" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37211 } } }
I20260927 18:39:08.874447 10110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:08.874495 10109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6 [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:08.875360 10113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:39:08.879894 10113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:39:08.888756 10113 catalog_manager.cc:1347] Generated new cluster ID: edb092d69afa4608bd4c5a6c4a4e8914
I20260927 18:39:08.888885 10113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:39:08.900776 10113 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:39:08.902724 10113 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:39:08.913655 10113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6: Generated new TSK 0
I20260927 18:39:08.914911 10113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab.
Entry for principal oryx/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab.
I20260927 18:39:08.978313 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37211
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab
--principal=oryx/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:09.316601 10129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:09.316898 10129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:09.317003 10129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:09.324944 10129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:09.325091 10129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:09.325177 10129 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:39:09.325258 10129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:39:09.335333 10129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab
--principal=oryx/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.95.126:37211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:09.336556 10129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:09.337921 10129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:09.350777 10134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:09.350845 10137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:09.350816 10135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:09.352017 10129 server_base.cc:1034] running on GCE node
I20260927 18:39:09.352676 10129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:09.353703 10129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:09.354965 10129 hybrid_clock.cc:648] HybridClock initialized: now 1790534349354877 us; error 54 us; skew 500 ppm
I20260927 18:39:09.355311 10129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:09 dist-test-slave-9jll krb5kdc[10013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534349, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:09.363608 10129 init.cc:377] Logged in from keytab as oryx/127.2.95.65@KRBTEST.COM (short username oryx)
I20260927 18:39:09.365226 10129 webserver.cc:460] Webserver started at http://127.2.95.65:46531/ using document root <none> and password file <none>
I20260927 18:39:09.366017 10129 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:09.366135 10129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:09.366511 10129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:09.369565 10129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "1ff04a7d37154373b757e1febc0b378a"
format_stamp: "Formatted at 2026-09-27 18:39:09 on dist-test-slave-9jll"
server_key: "6801e12220e162128bd61eddea79d871"
server_key_iv: "b551e95e6b3d6cec251c0167af453619"
server_key_version: "encryptionkey@0"
I20260927 18:39:09.370510 10129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "1ff04a7d37154373b757e1febc0b378a"
format_stamp: "Formatted at 2026-09-27 18:39:09 on dist-test-slave-9jll"
server_key: "6801e12220e162128bd61eddea79d871"
server_key_iv: "b551e95e6b3d6cec251c0167af453619"
server_key_version: "encryptionkey@0"
I20260927 18:39:09.376991 10129 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:39:09.380936 10144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:09.382566 10129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:39:09.382742 10129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "1ff04a7d37154373b757e1febc0b378a"
format_stamp: "Formatted at 2026-09-27 18:39:09 on dist-test-slave-9jll"
server_key: "6801e12220e162128bd61eddea79d871"
server_key_iv: "b551e95e6b3d6cec251c0167af453619"
server_key_version: "encryptionkey@0"
I20260927 18:39:09.382983 10129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:09.414698 10129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:09.419211 10129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:09.419478 10129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:09.422147 10129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:09.422245 10129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:09.422385 10129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:09.422446 10129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:39:09 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534349, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:09.446081 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:09.422918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:43383 (local address 127.2.95.126:37211)
0927 18:39:09.423421 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:09.423428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:09.424693 (+ 1265us) server_negotiation.cc:407] Connection header received
0927 18:39:09.427114 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:09.427120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:09.427218 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:09.427314 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:09.429653 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.430624 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.432926 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.433456 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.433592 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:09.438380 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:09.438389 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:09.438395 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:09.438446 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:09.442079 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:09.443068 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:09.443075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:09.443080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:09.443274 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:09.443818 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:09.443823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:09.443827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:09.444363 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:09.444531 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:09.445382 (+ 851us) server_negotiation.cc:299] Negotiation successful
0927 18:39:09.445694 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
I20260927 18:39:09.447254 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:09.423610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37211 (local address 127.2.95.65:43383)
0927 18:39:09.424485 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:09.424542 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:09.426767 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:09.427637 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:09.427653 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:09.428468 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:09.429433 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:09.429461 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.430842 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.430847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:09.432697 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:09.432719 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.433762 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.433771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:09.434033 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:09.434826 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:09.434852 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:09.438043 (+ 3191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:09.442350 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:09.442362 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:09.442384 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:09.442917 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:09.443429 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:09.443433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:09.443436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:09.443706 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:09.444554 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:09.444562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:09.444980 (+ 418us) client_negotiation.cc:769] Sending connection context
0927 18:39:09.445257 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 18:39:09.445746 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":368,"threads_started":1}
I20260927 18:39:09.468075 10129 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:33267
I20260927 18:39:09.468076 10264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:33267 every 8 connection(s)
I20260927 18:39:09.469271 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:09.454910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37211 (local address 127.2.95.65:59569)
0927 18:39:09.455542 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:09.455563 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:09.455731 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:09.456164 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:09.456170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:09.456765 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:09.457458 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:09.457471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.458868 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.458878 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:09.460181 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:09.460196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.461284 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.461292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:09.461459 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:09.462133 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:09.462154 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:09.463411 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:09.466724 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:09.466730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:09.466735 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:09.467225 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:09.467666 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:09.467672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:09.467675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:09.467804 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:09.468412 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:09.468417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:09.468572 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 18:39:09.468695 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 18:39:09.468949 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":335,"threads_started":1}
I20260927 18:39:09.469420 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:09.454860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:59569 (local address 127.2.95.126:37211)
0927 18:39:09.455074 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:09.455081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:09.455656 (+ 575us) server_negotiation.cc:407] Connection header received
0927 18:39:09.455896 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:09.455903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:09.455995 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:09.456118 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:09.457650 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.458684 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.460526 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.461091 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.461211 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:09.463642 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:09.463647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:09.463650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:09.463693 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:09.466575 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:09.467407 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:09.467413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:09.467417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:09.467552 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:09.467916 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:09.467921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:09.467924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:09.468233 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:09.468362 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:09.468856 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 18:39:09.469101 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:39:09.469637 10129 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:39:09.471017 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 10129
I20260927 18:39:09.471211 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:39:09.471865 2429 external_mini_cluster.cc:1428] Setting key 422bcb080acb4838a1fc34f7c053f25b
I20260927 18:39:09.473572 10265 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37211
I20260927 18:39:09.473821 10265 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:09.474611 10265 heartbeater.cc:511] Master 127.2.95.126:37211 requested a full tablet report, sending...
I20260927 18:39:09.476800 10058 ts_manager.cc:194] Registered new tserver with Master: 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267)
I20260927 18:39:09.478365 10058 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:43383
WARNING: no policy specified for oryx/127.2.95.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab.
Entry for principal oryx/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab.
I20260927 18:39:09.521565 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37211
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab
--principal=oryx/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:09.860306 10269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:09.860625 10269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:09.860740 10269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:09.868171 10269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:09.868297 10269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:09.868410 10269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:39:09.868518 10269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:39:09.879582 10269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab
--principal=oryx/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.95.126:37211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:09.881211 10269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:09.882814 10269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:09.898082 10275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:09.898229 10274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:09.898224 10277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:09.899194 10269 server_base.cc:1034] running on GCE node
I20260927 18:39:09.899843 10269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:09.900827 10269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:09.902125 10269 hybrid_clock.cc:648] HybridClock initialized: now 1790534349902053 us; error 35 us; skew 500 ppm
I20260927 18:39:09.902452 10269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:09 dist-test-slave-9jll krb5kdc[10013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534349, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:09.905854 10269 init.cc:377] Logged in from keytab as oryx/127.2.95.66@KRBTEST.COM (short username oryx)
I20260927 18:39:09.907457 10269 webserver.cc:460] Webserver started at http://127.2.95.66:38523/ using document root <none> and password file <none>
I20260927 18:39:09.908147 10269 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:09.908234 10269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:09.908497 10269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:09.911537 10269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "e6dccb7c09f34d348fa44bee9260a5e2"
format_stamp: "Formatted at 2026-09-27 18:39:09 on dist-test-slave-9jll"
server_key: "458da849963b241b7a10008147151822"
server_key_iv: "6c9c71dc29b2e4755b209f294bb0956d"
server_key_version: "encryptionkey@0"
I20260927 18:39:09.912388 10269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "e6dccb7c09f34d348fa44bee9260a5e2"
format_stamp: "Formatted at 2026-09-27 18:39:09 on dist-test-slave-9jll"
server_key: "458da849963b241b7a10008147151822"
server_key_iv: "6c9c71dc29b2e4755b209f294bb0956d"
server_key_version: "encryptionkey@0"
I20260927 18:39:09.917939 10269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 18:39:09.921849 10284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:09.923686 10269 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:39:09.923883 10269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "e6dccb7c09f34d348fa44bee9260a5e2"
format_stamp: "Formatted at 2026-09-27 18:39:09 on dist-test-slave-9jll"
server_key: "458da849963b241b7a10008147151822"
server_key_iv: "6c9c71dc29b2e4755b209f294bb0956d"
server_key_version: "encryptionkey@0"
I20260927 18:39:09.924235 10269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:09.943192 10269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:09.947862 10269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:09.948136 10269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:09.951190 10269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:09.951299 10269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:09.951439 10269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:09.951515 10269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:39:09 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534349, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:09.979673 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:09.952973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:60279 (local address 127.2.95.126:37211)
0927 18:39:09.953173 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:09.953179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:09.953455 (+ 276us) server_negotiation.cc:407] Connection header received
0927 18:39:09.955868 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:09.955876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:09.955979 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:09.956112 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:09.958653 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.959894 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.962830 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.963447 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.963603 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:09.969761 (+ 6158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:09.969771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:09.969777 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:09.969829 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:09.974195 (+ 4366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:09.976233 (+ 2038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:09.976242 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:09.976246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:09.976467 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:09.977278 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:09.977284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:09.977286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:09.977724 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:09.977907 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:09.979063 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0927 18:39:09.979321 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:39:09.981401 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:09.952361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37211 (local address 127.2.95.66:60279)
0927 18:39:09.953297 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:09.953336 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:09.955472 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:09.956303 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:09.956312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:09.957293 (+ 981us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:09.958292 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:09.958368 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.960130 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.960137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:09.962572 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:09.962633 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.963816 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.963826 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:09.964232 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:09.965241 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:09.965279 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:09.969313 (+ 4034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:09.974460 (+ 5147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:09.974472 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:09.974497 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:09.975795 (+ 1298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:09.976679 (+ 884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:09.976686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:09.976690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:09.977091 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:09.977885 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:09.977898 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:09.978436 (+ 538us) client_negotiation.cc:769] Sending connection context
0927 18:39:09.978805 (+ 369us) client_negotiation.cc:240] Negotiation successful
0927 18:39:09.979484 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":350,"threads_started":1}
I20260927 18:39:09.998610 10269 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:40781
I20260927 18:39:09.998642 10403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:40781 every 8 connection(s)
I20260927 18:39:09.999979 10269 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:39:10.003095 10404 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37211
I20260927 18:39:10.003334 10404 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:10.004189 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:09.988413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:52197 (local address 127.2.95.126:37211)
0927 18:39:09.988643 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:09.988648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:09.989321 (+ 673us) server_negotiation.cc:407] Connection header received
0927 18:39:09.989508 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:09.989521 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:09.989593 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:09.989700 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:09.990687 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.991402 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.992705 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.993110 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.993225 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:09.996999 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:09.997008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:09.997013 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:09.997062 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:09.999989 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:10.001079 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:10.001088 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:10.001092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:10.001460 (+ 368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:10.002121 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:10.002127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:10.002137 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:10.002587 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:10.002760 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:10.003391 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 18:39:10.003681 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260927 18:39:10.004317 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:09.988514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37211 (local address 127.2.95.66:52197)
0927 18:39:09.989195 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:09.989212 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:09.989379 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:09.989745 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:09.989750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:09.990237 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:09.990535 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:09.990544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.991540 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.991544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:09.992571 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:09.992583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:09.993478 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:09.993486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:09.993776 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:09.994554 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:09.994579 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:09.996612 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:10.000189 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:10.000196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:10.000200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:10.000858 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:10.001621 (+ 763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:10.001627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:10.001631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:10.001861 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:10.002732 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:10.002737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:10.003114 (+ 377us) client_negotiation.cc:769] Sending connection context
0927 18:39:10.003287 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 18:39:10.003665 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":405,"threads_started":1}
I20260927 18:39:10.004525 10404 heartbeater.cc:511] Master 127.2.95.126:37211 requested a full tablet report, sending...
I20260927 18:39:10.004884 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 10269
I20260927 18:39:10.005012 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:39:10.005502 2429 external_mini_cluster.cc:1428] Setting key 6fa78263bc110e31503a2aab6d3f3208
I20260927 18:39:10.008947 10058 ts_manager.cc:194] Registered new tserver with Master: e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66:40781)
I20260927 18:39:10.010036 10058 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:60279
WARNING: no policy specified for oryx/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab.
Entry for principal oryx/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab.
I20260927 18:39:10.058909 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37211
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab
--principal=oryx/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:10.393529 10408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:10.393877 10408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:10.394019 10408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:10.401355 10408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:10.401494 10408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:10.401582 10408 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 18:39:10.401690 10408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:39:10.411980 10408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab
--principal=oryx/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.95.126:37211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:10.413269 10408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:10.414631 10408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:10.428175 10414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:10.428201 10413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:10.428390 10416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:10.429095 10408 server_base.cc:1034] running on GCE node
I20260927 18:39:10.429811 10408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:10.430860 10408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:10.432122 10408 hybrid_clock.cc:648] HybridClock initialized: now 1790534350432031 us; error 42 us; skew 500 ppm
I20260927 18:39:10.432415 10408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:10 dist-test-slave-9jll krb5kdc[10013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534350, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:10.435658 10408 init.cc:377] Logged in from keytab as oryx/127.2.95.67@KRBTEST.COM (short username oryx)
I20260927 18:39:10.437212 10408 webserver.cc:460] Webserver started at http://127.2.95.67:37811/ using document root <none> and password file <none>
I20260927 18:39:10.438023 10408 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:10.438120 10408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:10.438392 10408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:10.441450 10408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "76a51c8b6eb343539951dcf9791675cb"
format_stamp: "Formatted at 2026-09-27 18:39:10 on dist-test-slave-9jll"
server_key: "96aae7984cdb132c6a5e1841d9d4c868"
server_key_iv: "266ff565dff2e402892fa434299e4d35"
server_key_version: "encryptionkey@0"
I20260927 18:39:10.442376 10408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "76a51c8b6eb343539951dcf9791675cb"
format_stamp: "Formatted at 2026-09-27 18:39:10 on dist-test-slave-9jll"
server_key: "96aae7984cdb132c6a5e1841d9d4c868"
server_key_iv: "266ff565dff2e402892fa434299e4d35"
server_key_version: "encryptionkey@0"
I20260927 18:39:10.448002 10408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:39:10.452023 10423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:10.453477 10408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:39:10.453711 10408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "76a51c8b6eb343539951dcf9791675cb"
format_stamp: "Formatted at 2026-09-27 18:39:10 on dist-test-slave-9jll"
server_key: "96aae7984cdb132c6a5e1841d9d4c868"
server_key_iv: "266ff565dff2e402892fa434299e4d35"
server_key_version: "encryptionkey@0"
I20260927 18:39:10.453954 10408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:10.470184 10408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:10.474412 10408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:10.474682 10408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:10.477226 10408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:10.477311 10408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:10.477439 10408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:10.477494 10408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:10.481547 10265 heartbeater.cc:503] Master 127.2.95.126:37211 was elected leader, sending a full tablet report...
Sep 27 18:39:10 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534350, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:10.501783 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.478091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:42157 (local address 127.2.95.126:37211)
0927 18:39:10.478279 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:10.478285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:10.479805 (+ 1520us) server_negotiation.cc:407] Connection header received
0927 18:39:10.482507 (+ 2702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:10.482515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:10.482604 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:10.482692 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:10.485553 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.486724 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.489209 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.489801 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.489953 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.494919 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:10.494927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:10.494932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:10.494991 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:10.497857 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:10.498677 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:10.498682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:10.498687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:10.498861 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:10.499488 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:10.499493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:10.499497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:10.499875 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:10.500041 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:10.501112 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0927 18:39:10.501367 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:39:10.502991 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.478697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37211 (local address 127.2.95.67:42157)
0927 18:39:10.479605 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:10.479656 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:10.482026 (+ 2370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:10.483543 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:10.483562 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:10.484385 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:10.485318 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.485356 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.486923 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.486928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.488961 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.488988 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.490118 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.490127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.490425 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.491284 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:10.491322 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:10.494653 (+ 3331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:10.498030 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:10.498042 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:10.498064 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:10.498543 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:10.499033 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:10.499038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:10.499041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:10.499320 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:10.500138 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:10.500147 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:10.500653 (+ 506us) client_negotiation.cc:769] Sending connection context
0927 18:39:10.501030 (+ 377us) client_negotiation.cc:240] Negotiation successful
0927 18:39:10.501658 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":395,"threads_started":1}
I20260927 18:39:10.524833 10408 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:33177
I20260927 18:39:10.524880 10542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:33177 every 8 connection(s)
I20260927 18:39:10.526743 10408 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:39:10.529330 10543 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37211
I20260927 18:39:10.529687 10543 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:10.530617 10543 heartbeater.cc:511] Master 127.2.95.126:37211 requested a full tablet report, sending...
I20260927 18:39:10.531291 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 10408
I20260927 18:39:10.531455 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:39:10.532017 2429 external_mini_cluster.cc:1428] Setting key bc80cdb266f139064074326bf3fee242
I20260927 18:39:10.533005 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.516636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37211 (local address 127.2.95.67:34623)
0927 18:39:10.517247 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:10.517268 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:10.517475 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:10.518171 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:10.518177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:10.519008 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:10.519560 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.519578 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.520800 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.520808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.522679 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.522700 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.523845 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.523853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.524076 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.524872 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:10.524896 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:10.526187 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:10.530306 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:10.530312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:10.530316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:10.530872 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:10.531370 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:10.531376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:10.531380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:10.531517 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:10.532166 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:10.532172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:10.532335 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 18:39:10.532452 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 18:39:10.532702 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":399,"threads_started":1}
I20260927 18:39:10.533226 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.516967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:34623 (local address 127.2.95.126:37211)
0927 18:39:10.517221 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:10.517228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:10.517363 (+ 135us) server_negotiation.cc:407] Connection header received
0927 18:39:10.517707 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:10.517713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:10.517815 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:10.517946 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:10.519770 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.520573 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.522945 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.523590 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.523717 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.526475 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:10.526481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:10.526485 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:10.526550 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:10.530128 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:10.531031 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:10.531036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:10.531040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:10.531233 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:10.531636 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:10.531640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:10.531644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:10.532017 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:10.532171 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:10.532627 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 18:39:10.532880 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 18:39:10.534214 10057 ts_manager.cc:194] Registered new tserver with Master: 76a51c8b6eb343539951dcf9791675cb (127.2.95.67:33177)
I20260927 18:39:10.535054 10057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:42157
I20260927 18:39:10.536970 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 18:39:10 dist-test-slave-9jll krb5kdc[10013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534350, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 18:39:10 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:10 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:39:10.588174 10153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:10.576882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.2.95.126:37211)
0927 18:39:10.577168 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:10.577173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:10.577359 (+ 186us) server_negotiation.cc:407] Connection header received
0927 18:39:10.577480 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:10.577485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:10.577553 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:10.577643 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:10.579059 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.579808 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.581162 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.581561 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.581691 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.587862 (+ 6171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50348: BlockingRecv error: recv got EOF from 127.0.0.1:50348 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 18:39:10 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534350, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:10.604796 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.591317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50362 (local address 127.2.95.126:37211)
0927 18:39:10.591524 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:10.591530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:10.591857 (+ 327us) server_negotiation.cc:407] Connection header received
0927 18:39:10.592103 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:10.592108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:10.592194 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:10.592309 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:10.593575 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.594411 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.595705 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.596120 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.596216 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.600093 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:10.600098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:10.600101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:10.600141 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:10.602560 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:10.603254 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:10.603258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:10.603260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:10.603379 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:10.603671 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:10.603674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:10.603676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:10.603967 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:10.604180 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:10.604399 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 18:39:10.604570 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 18:39:10.609480 10057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:39:10.616237 10057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:39:10.649494 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.629172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.2.95.66:40781)
0927 18:39:10.640095 (+ 10923us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:10.640103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:10.640129 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:39:10.640227 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:10.640235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:10.640504 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:10.640708 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:10.642764 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.644125 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.646498 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.647874 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.648018 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.648244 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:10.649019 (+ 775us) server_negotiation.cc:299] Negotiation successful
0927 18:39:10.649204 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10784,"thread_start_us":10651,"threads_started":1}
I20260927 18:39:10.649686 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.629499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:33177 (local address 127.0.0.1:35816)
0927 18:39:10.640020 (+ 10521us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:10.640039 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:10.640162 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:10.641204 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:10.641208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:10.641227 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:10.641841 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.641856 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.644018 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.644025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.646497 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.646511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.648471 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.648478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.648656 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.648789 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 18:39:10.648920 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 18:39:10.649172 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10396,"mutex_wait_us":159,"spinlock_wait_cycles":207104,"thread_start_us":9394,"threads_started":1}
I20260927 18:39:10.649703 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.629044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:40781 (local address 127.0.0.1:41444)
0927 18:39:10.639323 (+ 10279us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:10.639362 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:10.639549 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:10.640996 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:10.641005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:10.641041 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:10.642572 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.642585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.644285 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.644290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.646350 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.646360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.648407 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.648414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.648651 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.648790 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 18:39:10.648920 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 18:39:10.649172 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10003,"mutex_wait_us":139,"thread_start_us":10685,"threads_started":1}
I20260927 18:39:10.651404 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.630146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.2.95.67:33177)
0927 18:39:10.638618 (+ 8472us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:10.638630 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:10.640141 (+ 1511us) server_negotiation.cc:407] Connection header received
0927 18:39:10.640300 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:10.640308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:10.640560 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:10.640787 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:10.642107 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.643858 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.646661 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.648031 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.648140 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.648333 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:10.650885 (+ 2552us) server_negotiation.cc:299] Negotiation successful
0927 18:39:10.651080 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8216,"thread_start_us":8285,"threads_started":1}
I20260927 18:39:10.652029 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.640506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.2.95.65:33267)
0927 18:39:10.641947 (+ 1441us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:10.641956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:10.644640 (+ 2684us) server_negotiation.cc:407] Connection header received
0927 18:39:10.644815 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:10.644824 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:10.645117 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:10.645335 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:10.645974 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.647290 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.649452 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.651165 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.651296 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.651619 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:10.651709 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 18:39:10.651830 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":848,"threads_started":1}
I20260927 18:39:10.652136 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.640404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:33267 (local address 127.0.0.1:47604)
0927 18:39:10.644502 (+ 4098us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:10.644525 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:10.644658 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:10.645343 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:10.645348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:10.645374 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:10.645789 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.645799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.647435 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.647439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.649273 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.649286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.651371 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.651377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.651550 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.651582 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:39:10.651659 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:39:10.651789 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3958,"mutex_wait_us":3,"thread_start_us":3795,"threads_started":1}
I20260927 18:39:10.653527 10334 tablet_service.cc:1467] Processing CreateTablet for tablet 4bca041a666e4bf0961628e5e871c8dc (DEFAULT_TABLE table=test-table [id=613224ee6bfb4281b892a7c1804ba69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:10.654870 10334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bca041a666e4bf0961628e5e871c8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:10.655371 10195 tablet_service.cc:1467] Processing CreateTablet for tablet 4bca041a666e4bf0961628e5e871c8dc (DEFAULT_TABLE table=test-table [id=613224ee6bfb4281b892a7c1804ba69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:10.656437 10195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bca041a666e4bf0961628e5e871c8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:10.661307 10473 tablet_service.cc:1467] Processing CreateTablet for tablet 4bca041a666e4bf0961628e5e871c8dc (DEFAULT_TABLE table=test-table [id=613224ee6bfb4281b892a7c1804ba69f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:10.662846 10473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bca041a666e4bf0961628e5e871c8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:10.667641 10567 tablet_bootstrap.cc:492] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a: Bootstrap starting.
I20260927 18:39:10.672257 10567 tablet_bootstrap.cc:654] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:10.673777 10567 log.cc:824] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:10.675009 10568 tablet_bootstrap.cc:492] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2: Bootstrap starting.
I20260927 18:39:10.676846 10567 tablet_bootstrap.cc:492] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a: No bootstrap required, opened a new log
I20260927 18:39:10.677140 10567 ts_tablet_manager.cc:1397] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 18:39:10.678496 10570 tablet_bootstrap.cc:492] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb: Bootstrap starting.
I20260927 18:39:10.680635 10568 tablet_bootstrap.cc:654] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:10.681746 10570 tablet_bootstrap.cc:654] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:10.682000 10568 log.cc:824] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:10.682932 10570 log.cc:824] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:10.685425 10568 tablet_bootstrap.cc:492] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2: No bootstrap required, opened a new log
I20260927 18:39:10.685750 10568 ts_tablet_manager.cc:1397] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 18:39:10.687963 10570 tablet_bootstrap.cc:492] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb: No bootstrap required, opened a new log
I20260927 18:39:10.688228 10570 ts_tablet_manager.cc:1397] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 18:39:10.688376 10567 raft_consensus.cc:348] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:10.688848 10567 raft_consensus.cc:374] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:10.688947 10567 raft_consensus.cc:729] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff04a7d37154373b757e1febc0b378a, State: Initialized, Role: FOLLOWER
I20260927 18:39:10.689607 10567 consensus_queue.cc:260] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:10.695729 10567 ts_tablet_manager.cc:1428] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a: Time spent starting tablet: real 0.018s user 0.013s sys 0.000s
I20260927 18:39:10.696085 10568 raft_consensus.cc:348] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:10.696677 10568 raft_consensus.cc:374] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:10.696789 10568 raft_consensus.cc:729] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dccb7c09f34d348fa44bee9260a5e2, State: Initialized, Role: FOLLOWER
I20260927 18:39:10.696799 10570 raft_consensus.cc:348] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:10.697177 10570 raft_consensus.cc:374] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:10.697281 10570 raft_consensus.cc:729] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a51c8b6eb343539951dcf9791675cb, State: Initialized, Role: FOLLOWER
I20260927 18:39:10.697468 10568 consensus_queue.cc:260] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:10.697856 10570 consensus_queue.cc:260] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:10.698611 10404 heartbeater.cc:503] Master 127.2.95.126:37211 was elected leader, sending a full tablet report...
I20260927 18:39:10.699443 10568 ts_tablet_manager.cc:1428] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260927 18:39:10.700655 10543 heartbeater.cc:503] Master 127.2.95.126:37211 was elected leader, sending a full tablet report...
I20260927 18:39:10.701162 10570 ts_tablet_manager.cc:1428] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260927 18:39:10.722350 10266 tablet.cc:2369] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:39:10.753018 10405 tablet.cc:2369] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:39:10.778708 10544 tablet.cc:2369] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:39:10.952903 10574 raft_consensus.cc:482] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:39:10.953228 10574 raft_consensus.cc:504] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:10.955037 10574 leader_election.cc:288] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267), 76a51c8b6eb343539951dcf9791675cb (127.2.95.67:33177)
I20260927 18:39:10.963567 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.955578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:33719 (local address 127.2.95.67:33177)
0927 18:39:10.955764 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:10.955771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:10.956554 (+ 783us) server_negotiation.cc:407] Connection header received
0927 18:39:10.956729 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:10.956733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:10.956788 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:10.956867 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:10.957531 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.958325 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.961104 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.962359 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.962557 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.962663 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:10.963145 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 18:39:10.963269 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:39:10.963574 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.955469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:35193 (local address 127.2.95.65:33267)
0927 18:39:10.955682 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:10.955688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:10.956500 (+ 812us) server_negotiation.cc:407] Connection header received
0927 18:39:10.956684 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:10.956690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:10.956772 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:10.956841 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:10.957568 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.958353 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.961104 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.962427 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.962556 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.962663 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:10.963244 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 18:39:10.963368 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:39:10.963703 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.955480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:33267 (local address 127.2.95.66:35193)
0927 18:39:10.956355 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:10.956372 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:10.956535 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:10.956875 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:10.956880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:10.956911 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:10.957422 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.957432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.958534 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.958537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.960904 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.960913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.962679 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.962684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.962963 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.963121 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 18:39:10.963193 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 18:39:10.963297 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":185,"thread_start_us":399,"threads_started":1}
I20260927 18:39:10.964150 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:10.955540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:33177 (local address 127.2.95.66:33719)
0927 18:39:10.956423 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:10.956443 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:10.956591 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:10.956898 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:10.956902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:10.956925 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:10.957372 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.957385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.958467 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.958472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.960895 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:10.960912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:10.962678 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:10.962684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:10.962944 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:10.963021 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 18:39:10.963121 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 18:39:10.963269 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":72,"thread_start_us":559,"threads_started":1}
I20260927 18:39:10.964716 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bca041a666e4bf0961628e5e871c8dc" candidate_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff04a7d37154373b757e1febc0b378a" is_pre_election: true
I20260927 18:39:10.964974 10493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bca041a666e4bf0961628e5e871c8dc" candidate_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a51c8b6eb343539951dcf9791675cb" is_pre_election: true
I20260927 18:39:10.965322 10217 raft_consensus.cc:2454] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dccb7c09f34d348fa44bee9260a5e2 in term 0.
I20260927 18:39:10.965392 10493 raft_consensus.cc:2454] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dccb7c09f34d348fa44bee9260a5e2 in term 0.
I20260927 18:39:10.965999 10288 leader_election.cc:302] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ff04a7d37154373b757e1febc0b378a, 76a51c8b6eb343539951dcf9791675cb, e6dccb7c09f34d348fa44bee9260a5e2; no voters:
I20260927 18:39:10.966599 10574 raft_consensus.cc:2781] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:39:10.966753 10574 raft_consensus.cc:482] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:39:10.966861 10574 raft_consensus.cc:3037] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:10.969041 10574 raft_consensus.cc:504] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:10.969611 10574 leader_election.cc:288] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [CANDIDATE]: Term 1 election: Requested vote from peers 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267), 76a51c8b6eb343539951dcf9791675cb (127.2.95.67:33177)
I20260927 18:39:10.970228 10493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bca041a666e4bf0961628e5e871c8dc" candidate_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a51c8b6eb343539951dcf9791675cb"
I20260927 18:39:10.970216 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bca041a666e4bf0961628e5e871c8dc" candidate_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff04a7d37154373b757e1febc0b378a"
I20260927 18:39:10.970414 10493 raft_consensus.cc:3037] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:10.970418 10217 raft_consensus.cc:3037] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:10.972662 10493 raft_consensus.cc:2454] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dccb7c09f34d348fa44bee9260a5e2 in term 1.
I20260927 18:39:10.972720 10217 raft_consensus.cc:2454] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dccb7c09f34d348fa44bee9260a5e2 in term 1.
I20260927 18:39:10.973109 10288 leader_election.cc:302] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76a51c8b6eb343539951dcf9791675cb, e6dccb7c09f34d348fa44bee9260a5e2; no voters:
I20260927 18:39:10.973434 10574 raft_consensus.cc:2781] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:10.974002 10574 raft_consensus.cc:686] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 LEADER]: Becoming Leader. State: Replica: e6dccb7c09f34d348fa44bee9260a5e2, State: Running, Role: LEADER
I20260927 18:39:10.974543 10574 consensus_queue.cc:237] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:10.978909 10058 catalog_manager.cc:5495] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66). New cstate: current_term: 1 leader_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } health_report { overall_health: UNKNOWN } } }
I20260927 18:39:11.009775 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.004019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.2.95.66:40781)
0927 18:39:11.004224 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:11.004231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:11.004417 (+ 186us) server_negotiation.cc:407] Connection header received
0927 18:39:11.004729 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:11.004733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:11.004810 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:11.004956 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:39:11.005342 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.006161 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.007493 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.007938 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.008067 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.008511 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:39:11.009032 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:39:11.009161 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:11.009263 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 18:39:11.009346 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:39:11.018131 10493 raft_consensus.cc:1261] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Refusing update from remote peer e6dccb7c09f34d348fa44bee9260a5e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:11.018103 10217 raft_consensus.cc:1261] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Refusing update from remote peer e6dccb7c09f34d348fa44bee9260a5e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:11.019362 10578 consensus_queue.cc:1035] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:11.019738 10574 consensus_queue.cc:1035] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:11.046095 10057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:11.046373 10057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:11.050292 10057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6: Sending DeleteTablet for 3 replicas of tablet 4bca041a666e4bf0961628e5e871c8dc
I20260927 18:39:11.051592 10473 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bca041a666e4bf0961628e5e871c8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:11 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:35816
I20260927 18:39:11.051787 10334 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bca041a666e4bf0961628e5e871c8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:11 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:41444
I20260927 18:39:11.051888 10195 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bca041a666e4bf0961628e5e871c8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:11 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:47604
I20260927 18:39:11.053517 10057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:39:11.054273 10057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:39:11.065111 10194 tablet_service.cc:1467] Processing CreateTablet for tablet fde342f451b44f20b8e32edc45a6ecb8 (DEFAULT_TABLE table=test-table [id=4699507213f94e0d8fb49d41695a7199]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:11.065124 10472 tablet_service.cc:1467] Processing CreateTablet for tablet fde342f451b44f20b8e32edc45a6ecb8 (DEFAULT_TABLE table=test-table [id=4699507213f94e0d8fb49d41695a7199]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:11.065303 10333 tablet_service.cc:1467] Processing CreateTablet for tablet fde342f451b44f20b8e32edc45a6ecb8 (DEFAULT_TABLE table=test-table [id=4699507213f94e0d8fb49d41695a7199]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:11.065632 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde342f451b44f20b8e32edc45a6ecb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:11.065743 10194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde342f451b44f20b8e32edc45a6ecb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:11.066107 10333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde342f451b44f20b8e32edc45a6ecb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:11.072532 10570 tablet_bootstrap.cc:492] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb: Bootstrap starting.
I20260927 18:39:11.072745 10567 tablet_bootstrap.cc:492] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a: Bootstrap starting.
I20260927 18:39:11.074291 10570 tablet_bootstrap.cc:654] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:11.074431 10567 tablet_bootstrap.cc:654] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:11.077322 10568 tablet_bootstrap.cc:492] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2: Bootstrap starting.
I20260927 18:39:11.078899 10593 tablet_replica.cc:330] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a: stopping tablet replica
I20260927 18:39:11.079460 10568 tablet_bootstrap.cc:654] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:11.079461 10593 raft_consensus.cc:2229] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:11.079988 10593 raft_consensus.cc:2258] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:11.084782 10570 tablet_bootstrap.cc:492] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb: No bootstrap required, opened a new log
I20260927 18:39:11.085150 10570 ts_tablet_manager.cc:1397] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20260927 18:39:11.085150 10593 ts_tablet_manager.cc:1905] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:11.086220 10570 raft_consensus.cc:348] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.086582 10570 raft_consensus.cc:374] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:11.086777 10570 raft_consensus.cc:729] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a51c8b6eb343539951dcf9791675cb, State: Initialized, Role: FOLLOWER
I20260927 18:39:11.087199 10570 consensus_queue.cc:260] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.090695 10570 ts_tablet_manager.cc:1428] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 18:39:11.092275 10593 ts_tablet_manager.cc:1918] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:11.092487 10593 log.cc:1196] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/4bca041a666e4bf0961628e5e871c8dc
I20260927 18:39:11.093031 10567 tablet_bootstrap.cc:492] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a: No bootstrap required, opened a new log
I20260927 18:39:11.093168 10593 ts_tablet_manager.cc:1939] T 4bca041a666e4bf0961628e5e871c8dc P 1ff04a7d37154373b757e1febc0b378a: Deleting consensus metadata
I20260927 18:39:11.093386 10567 ts_tablet_manager.cc:1397] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20260927 18:39:11.094161 10592 tablet_replica.cc:330] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2: stopping tablet replica
I20260927 18:39:11.094635 10592 raft_consensus.cc:2229] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:39:11.094656 10567 raft_consensus.cc:348] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.095043 10567 raft_consensus.cc:374] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:11.095173 10567 raft_consensus.cc:729] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff04a7d37154373b757e1febc0b378a, State: Initialized, Role: FOLLOWER
I20260927 18:39:11.095295 10592 raft_consensus.cc:2258] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:11.095579 10567 consensus_queue.cc:260] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.096175 10567 ts_tablet_manager.cc:1428] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 18:39:11.097478 10592 ts_tablet_manager.cc:1905] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:11.097847 10043 catalog_manager.cc:4841] TS 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267): tablet 4bca041a666e4bf0961628e5e871c8dc (table test-table [id=613224ee6bfb4281b892a7c1804ba69f]) successfully deleted
I20260927 18:39:11.098668 10591 tablet_replica.cc:330] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb: stopping tablet replica
I20260927 18:39:11.099164 10591 raft_consensus.cc:2229] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:11.099697 10591 raft_consensus.cc:2258] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:11.101426 10568 tablet_bootstrap.cc:492] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2: No bootstrap required, opened a new log
I20260927 18:39:11.101692 10568 ts_tablet_manager.cc:1397] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.000s
I20260927 18:39:11.102002 10591 ts_tablet_manager.cc:1905] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:11.102665 10568 raft_consensus.cc:348] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.102979 10568 raft_consensus.cc:374] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:11.103111 10568 raft_consensus.cc:729] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dccb7c09f34d348fa44bee9260a5e2, State: Initialized, Role: FOLLOWER
I20260927 18:39:11.103451 10568 consensus_queue.cc:260] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.104190 10568 ts_tablet_manager.cc:1428] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 18:39:11.108305 10592 ts_tablet_manager.cc:1918] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:11.108507 10592 log.cc:1196] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/4bca041a666e4bf0961628e5e871c8dc
I20260927 18:39:11.109074 10592 ts_tablet_manager.cc:1939] T 4bca041a666e4bf0961628e5e871c8dc P e6dccb7c09f34d348fa44bee9260a5e2: Deleting consensus metadata
I20260927 18:39:11.109349 10591 ts_tablet_manager.cc:1918] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:11.109500 10591 log.cc:1196] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/4bca041a666e4bf0961628e5e871c8dc
I20260927 18:39:11.110065 10591 ts_tablet_manager.cc:1939] T 4bca041a666e4bf0961628e5e871c8dc P 76a51c8b6eb343539951dcf9791675cb: Deleting consensus metadata
I20260927 18:39:11.111099 10043 catalog_manager.cc:4841] TS e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66:40781): tablet 4bca041a666e4bf0961628e5e871c8dc (table test-table [id=613224ee6bfb4281b892a7c1804ba69f]) successfully deleted
I20260927 18:39:11.111483 10045 catalog_manager.cc:4841] TS 76a51c8b6eb343539951dcf9791675cb (127.2.95.67:33177): tablet 4bca041a666e4bf0961628e5e871c8dc (table test-table [id=613224ee6bfb4281b892a7c1804ba69f]) successfully deleted
I20260927 18:39:11.254557 10586 raft_consensus.cc:482] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:39:11.254786 10586 raft_consensus.cc:504] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.255586 10586 leader_election.cc:288] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267), 76a51c8b6eb343539951dcf9791675cb (127.2.95.67:33177)
I20260927 18:39:11.256160 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde342f451b44f20b8e32edc45a6ecb8" candidate_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff04a7d37154373b757e1febc0b378a" is_pre_election: true
I20260927 18:39:11.256152 10493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde342f451b44f20b8e32edc45a6ecb8" candidate_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a51c8b6eb343539951dcf9791675cb" is_pre_election: true
I20260927 18:39:11.256407 10217 raft_consensus.cc:2454] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dccb7c09f34d348fa44bee9260a5e2 in term 0.
I20260927 18:39:11.256407 10493 raft_consensus.cc:2454] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6dccb7c09f34d348fa44bee9260a5e2 in term 0.
I20260927 18:39:11.256863 10286 leader_election.cc:302] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff04a7d37154373b757e1febc0b378a, e6dccb7c09f34d348fa44bee9260a5e2; no voters:
I20260927 18:39:11.257184 10586 raft_consensus.cc:2781] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:39:11.257325 10586 raft_consensus.cc:482] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:39:11.257418 10586 raft_consensus.cc:3037] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:11.260607 10586 raft_consensus.cc:504] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.261327 10586 leader_election.cc:288] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [CANDIDATE]: Term 1 election: Requested vote from peers 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267), 76a51c8b6eb343539951dcf9791675cb (127.2.95.67:33177)
I20260927 18:39:11.261931 10493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde342f451b44f20b8e32edc45a6ecb8" candidate_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a51c8b6eb343539951dcf9791675cb"
I20260927 18:39:11.261967 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde342f451b44f20b8e32edc45a6ecb8" candidate_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff04a7d37154373b757e1febc0b378a"
I20260927 18:39:11.262190 10217 raft_consensus.cc:3037] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:11.262203 10493 raft_consensus.cc:3037] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:11.264140 10493 raft_consensus.cc:2454] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dccb7c09f34d348fa44bee9260a5e2 in term 1.
I20260927 18:39:11.264598 10288 leader_election.cc:302] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76a51c8b6eb343539951dcf9791675cb, e6dccb7c09f34d348fa44bee9260a5e2; no voters:
I20260927 18:39:11.264896 10586 raft_consensus.cc:2781] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:11.265051 10586 raft_consensus.cc:686] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 LEADER]: Becoming Leader. State: Replica: e6dccb7c09f34d348fa44bee9260a5e2, State: Running, Role: LEADER
I20260927 18:39:11.265034 10217 raft_consensus.cc:2454] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6dccb7c09f34d348fa44bee9260a5e2 in term 1.
I20260927 18:39:11.265347 10586 consensus_queue.cc:237] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.268144 10058 catalog_manager.cc:5495] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66). New cstate: current_term: 1 leader_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } health_report { overall_health: UNKNOWN } } }
Sep 27 18:39:11 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534348, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.126@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:11.332579 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.315992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37211 (local address 127.0.0.1:50366)
0927 18:39:11.316762 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:11.316785 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:11.316952 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:11.317474 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:11.317478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:11.318307 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:11.318560 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:11.318570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.320072 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.320076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:11.321134 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:11.321146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.322066 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.322071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:11.322199 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.322900 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:11.322923 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:11.325535 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:11.329512 (+ 3977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:11.329530 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:11.329539 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:11.330163 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:11.330694 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:11.330700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:11.330703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:11.330925 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:11.331569 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:11.331576 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:11.331737 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 18:39:11.332038 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 18:39:11.332279 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":391,"threads_started":1}
I20260927 18:39:11.333108 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.316058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.2.95.126:37211)
0927 18:39:11.317132 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:11.317143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:11.317168 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:39:11.317225 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:11.317230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:11.317303 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:11.317423 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:11.318722 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.319908 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.321269 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.321910 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.322033 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.325834 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:11.325841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:11.325846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:11.325886 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:11.329316 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:11.330334 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:11.330340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:11.330344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:11.330527 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:11.331049 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:11.331054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:11.331057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:11.331413 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:11.331585 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:11.332157 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 18:39:11.332448 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":887,"thread_start_us":744,"threads_started":1}
I20260927 18:39:11.338619 10058 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:50366:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 18:39:11.339908 10058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:39:11.347043 10194 tablet_service.cc:1467] Processing CreateTablet for tablet b6601952fb43472c929bbdc47101b37c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b0ee3c208a548bba8fa6d6b87743cf9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:39:11.347200 10473 tablet_service.cc:1467] Processing CreateTablet for tablet b6601952fb43472c929bbdc47101b37c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b0ee3c208a548bba8fa6d6b87743cf9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:39:11.347653 10194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6601952fb43472c929bbdc47101b37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:11.347678 10473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6601952fb43472c929bbdc47101b37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:11.353626 10570 tablet_bootstrap.cc:492] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: Bootstrap starting.
I20260927 18:39:11.354249 10334 tablet_service.cc:1467] Processing CreateTablet for tablet b6601952fb43472c929bbdc47101b37c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b0ee3c208a548bba8fa6d6b87743cf9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 18:39:11.354833 10334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6601952fb43472c929bbdc47101b37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:11.355036 10570 tablet_bootstrap.cc:654] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:11.357046 10570 tablet_bootstrap.cc:492] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: No bootstrap required, opened a new log
I20260927 18:39:11.357214 10570 ts_tablet_manager.cc:1397] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 18:39:11.357883 10570 raft_consensus.cc:348] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.358407 10570 raft_consensus.cc:374] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:11.358507 10570 raft_consensus.cc:729] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a51c8b6eb343539951dcf9791675cb, State: Initialized, Role: FOLLOWER
I20260927 18:39:11.359454 10570 consensus_queue.cc:260] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.360064 10570 ts_tablet_manager.cc:1428] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:39:11.359948 10575 tablet_replica.cc:439] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } } }
I20260927 18:39:11.360291 10575 tablet_replica.cc:442] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:39:11.361186 10567 tablet_bootstrap.cc:492] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a: Bootstrap starting.
I20260927 18:39:11.362412 10568 tablet_bootstrap.cc:492] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2: Bootstrap starting.
I20260927 18:39:11.362949 10567 tablet_bootstrap.cc:654] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:11.364087 10568 tablet_bootstrap.cc:654] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:11.366097 10567 tablet_bootstrap.cc:492] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a: No bootstrap required, opened a new log
I20260927 18:39:11.366247 10567 ts_tablet_manager.cc:1397] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260927 18:39:11.366956 10567 raft_consensus.cc:348] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.367206 10567 raft_consensus.cc:374] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:11.367277 10568 tablet_bootstrap.cc:492] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2: No bootstrap required, opened a new log
I20260927 18:39:11.367295 10567 raft_consensus.cc:729] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff04a7d37154373b757e1febc0b378a, State: Initialized, Role: FOLLOWER
I20260927 18:39:11.367466 10568 ts_tablet_manager.cc:1397] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260927 18:39:11.367518 10567 consensus_queue.cc:260] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.368602 10568 raft_consensus.cc:348] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.368855 10568 raft_consensus.cc:374] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:11.368956 10568 raft_consensus.cc:729] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dccb7c09f34d348fa44bee9260a5e2, State: Initialized, Role: FOLLOWER
I20260927 18:39:11.369333 10567 ts_tablet_manager.cc:1428] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:39:11.369246 10568 consensus_queue.cc:260] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.369908 10568 ts_tablet_manager.cc:1428] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:39:11.369755 10586 tablet_replica.cc:439] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } } }
I20260927 18:39:11.370077 10586 tablet_replica.cc:442] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:39:11.368021 10573 tablet_replica.cc:439] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } } }
I20260927 18:39:11.370462 10573 tablet_replica.cc:442] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:39:11.455164 10575 raft_consensus.cc:482] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:39:11.455410 10575 raft_consensus.cc:504] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.456971 10575 leader_election.cc:288] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267), e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66:40781)
I20260927 18:39:11.465370 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.457315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:39751 (local address 127.2.95.65:33267)
0927 18:39:11.457521 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:11.457528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:11.458086 (+ 558us) server_negotiation.cc:407] Connection header received
0927 18:39:11.458334 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:11.458342 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:11.458454 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:11.458523 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:11.459309 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.460307 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.462896 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.464169 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.464301 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.464413 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:11.464950 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 18:39:11.465096 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:39:11.465409 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.457306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:33267 (local address 127.2.95.67:39751)
0927 18:39:11.457981 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:11.458001 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:11.458145 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:11.458821 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:11.458827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:11.458855 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:11.459145 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:11.459157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.460482 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.460488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:11.462688 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:11.462705 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.464504 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.464510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:11.464696 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.464736 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:39:11.464899 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 18:39:11.465047 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":23,"thread_start_us":393,"threads_started":1}
I20260927 18:39:11.465557 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.458320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:44169 (local address 127.2.95.66:40781)
0927 18:39:11.458484 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:11.458489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:11.459157 (+ 668us) server_negotiation.cc:407] Connection header received
0927 18:39:11.459387 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:11.459392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:11.459480 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:11.459571 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:11.460210 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.461307 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.463414 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.464563 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.464699 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.464786 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:11.465151 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 18:39:11.465296 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:39:11.466333 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.458154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:40781 (local address 127.2.95.67:44169)
0927 18:39:11.459019 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:11.459041 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:11.459201 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:11.459598 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:11.459603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:11.459627 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:11.460020 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:11.460031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.461440 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.461446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:11.463264 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:11.463275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.464772 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.464778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:11.464952 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.464993 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:39:11.465740 (+ 747us) client_negotiation.cc:240] Negotiation successful
0927 18:39:11.465889 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":1,"thread_start_us":486,"threads_started":1}
I20260927 18:39:11.466427 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6601952fb43472c929bbdc47101b37c" candidate_uuid: "76a51c8b6eb343539951dcf9791675cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff04a7d37154373b757e1febc0b378a" is_pre_election: true
I20260927 18:39:11.466704 10217 raft_consensus.cc:2454] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76a51c8b6eb343539951dcf9791675cb in term 0.
I20260927 18:39:11.467134 10354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6601952fb43472c929bbdc47101b37c" candidate_uuid: "76a51c8b6eb343539951dcf9791675cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" is_pre_election: true
I20260927 18:39:11.467213 10425 leader_election.cc:302] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff04a7d37154373b757e1febc0b378a, 76a51c8b6eb343539951dcf9791675cb; no voters:
I20260927 18:39:11.467530 10354 raft_consensus.cc:2454] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76a51c8b6eb343539951dcf9791675cb in term 0.
I20260927 18:39:11.467622 10575 raft_consensus.cc:2781] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:39:11.467721 10575 raft_consensus.cc:482] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:39:11.467803 10575 raft_consensus.cc:3037] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:11.470014 10575 raft_consensus.cc:504] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.470595 10575 leader_election.cc:288] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [CANDIDATE]: Term 1 election: Requested vote from peers 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267), e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66:40781)
I20260927 18:39:11.471175 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6601952fb43472c929bbdc47101b37c" candidate_uuid: "76a51c8b6eb343539951dcf9791675cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff04a7d37154373b757e1febc0b378a"
I20260927 18:39:11.471383 10217 raft_consensus.cc:3037] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:11.473619 10217 raft_consensus.cc:2454] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a51c8b6eb343539951dcf9791675cb in term 1.
I20260927 18:39:11.474159 10425 leader_election.cc:302] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff04a7d37154373b757e1febc0b378a, 76a51c8b6eb343539951dcf9791675cb; no voters:
I20260927 18:39:11.474308 10354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6601952fb43472c929bbdc47101b37c" candidate_uuid: "76a51c8b6eb343539951dcf9791675cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dccb7c09f34d348fa44bee9260a5e2"
I20260927 18:39:11.474504 10575 raft_consensus.cc:2781] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:11.474548 10354 raft_consensus.cc:3037] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:11.477864 10354 raft_consensus.cc:2454] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a51c8b6eb343539951dcf9791675cb in term 1.
I20260927 18:39:11.478803 10575 raft_consensus.cc:686] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [term 1 LEADER]: Becoming Leader. State: Replica: 76a51c8b6eb343539951dcf9791675cb, State: Running, Role: LEADER
I20260927 18:39:11.479187 10575 consensus_queue.cc:237] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.481165 10610 tablet_replica.cc:439] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: TxnStatusTablet state changed. Reason: New leader 76a51c8b6eb343539951dcf9791675cb. Latest consensus state: current_term: 1 leader_uuid: "76a51c8b6eb343539951dcf9791675cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } } }
I20260927 18:39:11.481547 10610 tablet_replica.cc:442] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:39:11.484486 10057 catalog_manager.cc:5495] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 76a51c8b6eb343539951dcf9791675cb (127.2.95.67). New cstate: current_term: 1 leader_uuid: "76a51c8b6eb343539951dcf9791675cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } health_report { overall_health: HEALTHY } } }
I20260927 18:39:11.487468 10612 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 18:39:11.487749 10612 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 18:39:11.529553 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.523036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:33177 (local address 127.0.0.1:35818)
0927 18:39:11.523308 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:11.523332 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:11.523503 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:11.524472 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:11.524477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:11.524490 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:39:11.524700 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:11.524711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.525950 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.525956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:11.526979 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:11.526993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.527767 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.527774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:11.527928 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.527990 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:39:11.528954 (+ 964us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:39:11.528985 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:39:11.529066 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:39:11.529231 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 18:39:11.529619 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.523049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.2.95.67:33177)
0927 18:39:11.524088 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:11.524093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:11.524113 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:39:11.524199 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:11.524203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:11.524288 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:11.524433 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:39:11.524858 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.525809 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.527136 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.527606 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.527723 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.528216 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:39:11.528839 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:39:11.528979 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:11.529132 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 18:39:11.529214 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":748,"threads_started":1}
I20260927 18:39:11.533941 10217 raft_consensus.cc:1261] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Refusing update from remote peer 76a51c8b6eb343539951dcf9791675cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:11.534276 10354 raft_consensus.cc:1261] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Refusing update from remote peer 76a51c8b6eb343539951dcf9791675cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:11.534767 10610 consensus_queue.cc:1035] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:11.535106 10575 consensus_queue.cc:1035] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:11.538606 10586 tablet_replica.cc:439] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2: TxnStatusTablet state changed. Reason: New leader 76a51c8b6eb343539951dcf9791675cb. Latest consensus state: current_term: 1 leader_uuid: "76a51c8b6eb343539951dcf9791675cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } } }
I20260927 18:39:11.538911 10586 tablet_replica.cc:442] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:39:11.541357 10575 tablet_replica.cc:439] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76a51c8b6eb343539951dcf9791675cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } } }
I20260927 18:39:11.541703 10575 tablet_replica.cc:442] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:39:11.542990 10586 tablet_replica.cc:439] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76a51c8b6eb343539951dcf9791675cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } } }
I20260927 18:39:11.543263 10586 tablet_replica.cc:442] T b6601952fb43472c929bbdc47101b37c P e6dccb7c09f34d348fa44bee9260a5e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:39:11.555882 10573 tablet_replica.cc:439] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a: TxnStatusTablet state changed. Reason: New leader 76a51c8b6eb343539951dcf9791675cb. Latest consensus state: current_term: 1 leader_uuid: "76a51c8b6eb343539951dcf9791675cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } } }
I20260927 18:39:11.556164 10573 tablet_replica.cc:442] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:39:11.557651 10610 tablet_replica.cc:439] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76a51c8b6eb343539951dcf9791675cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } } }
I20260927 18:39:11.557976 10610 tablet_replica.cc:442] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb: This TxnStatusTablet replica's current role is: LEADER
I20260927 18:39:11.559290 10616 mvcc.cc:205] Tried to move back new op lower bound from 7334028703875690496 to 7334028703662612480. Current Snapshot: MvccSnapshot[applied={T|T < 7334028703875690496}]
I20260927 18:39:11.568037 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.562740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:60835 (local address 127.2.95.67:33177)
0927 18:39:11.563065 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:11.563071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:11.563475 (+ 404us) server_negotiation.cc:407] Connection header received
0927 18:39:11.563665 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:11.563671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:11.563739 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:11.563814 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:39:11.564273 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.565019 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.566308 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.566801 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.566913 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.567307 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:39:11.567413 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:39:11.567487 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:11.567615 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 18:39:11.567701 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260927 18:39:11.568095 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.562738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:33177 (local address 127.2.95.66:60835)
0927 18:39:11.563377 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:11.563393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:11.563532 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:11.563835 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:11.563840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:11.563851 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:39:11.564139 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:11.564148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.565145 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.565150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:11.566180 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:11.566193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.566947 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.566953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:11.567090 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.567147 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:39:11.567501 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:39:11.567520 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 18:39:11.567586 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:39:11.567722 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":337,"threads_started":1}
I20260927 18:39:11.568645 10573 tablet_replica.cc:439] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76a51c8b6eb343539951dcf9791675cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } } }
I20260927 18:39:11.568885 10573 tablet_replica.cc:442] T b6601952fb43472c929bbdc47101b37c P 1ff04a7d37154373b757e1febc0b378a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 18:39:11.580253 10493 raft_consensus.cc:1261] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Refusing update from remote peer e6dccb7c09f34d348fa44bee9260a5e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:11.580674 10215 raft_consensus.cc:1261] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Refusing update from remote peer e6dccb7c09f34d348fa44bee9260a5e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:11.581022 10586 consensus_queue.cc:1035] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:11.581346 10574 consensus_queue.cc:1035] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:11.597007 10617 mvcc.cc:205] Tried to move back new op lower bound from 7334028704065863680 to 7334028702785245184. Current Snapshot: MvccSnapshot[applied={T|T < 7334028704065863680}]
I20260927 18:39:11.598882 10582 mvcc.cc:205] Tried to move back new op lower bound from 7334028704065863680 to 7334028702785245184. Current Snapshot: MvccSnapshot[applied={T|T < 7334028704065863680}]
I20260927 18:39:11.627466 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.620398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:56611 (local address 127.2.95.66:40781)
0927 18:39:11.620761 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:11.620768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:11.621466 (+ 698us) server_negotiation.cc:407] Connection header received
0927 18:39:11.621663 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:11.621689 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:11.621776 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:11.621876 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:39:11.622806 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.623892 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.625601 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.626128 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.626243 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.626757 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:39:11.626898 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:39:11.626948 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:11.627127 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 18:39:11.627204 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260927 18:39:11.627629 10631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.620424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:40781 (local address 127.2.95.67:56611)
0927 18:39:11.621347 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:11.621371 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:11.621518 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:11.622272 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:11.622276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:11.622289 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:39:11.622653 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:11.622668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.624021 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.624025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:11.625424 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:11.625442 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.626314 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.626320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:11.626458 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.626546 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:39:11.627013 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:39:11.627020 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:39:11.627093 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 18:39:11.627251 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":559,"threads_started":1}
I20260927 18:39:11.663686 10057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:11.664004 10057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:11.667631 10057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6: Sending DeleteTablet for 3 replicas of tablet fde342f451b44f20b8e32edc45a6ecb8
I20260927 18:39:11.668599 10194 tablet_service.cc:1514] Processing DeleteTablet for tablet fde342f451b44f20b8e32edc45a6ecb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:11 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:47604
I20260927 18:39:11.668574 10334 tablet_service.cc:1514] Processing DeleteTablet for tablet fde342f451b44f20b8e32edc45a6ecb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:11 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:41444
I20260927 18:39:11.669013 10473 tablet_service.cc:1514] Processing DeleteTablet for tablet fde342f451b44f20b8e32edc45a6ecb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:11 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:35816
I20260927 18:39:11.679701 10634 tablet_replica.cc:330] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a: stopping tablet replica
I20260927 18:39:11.679951 10634 raft_consensus.cc:2229] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:11.680197 10634 raft_consensus.cc:2258] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:11.680727 10633 tablet_replica.cc:330] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2: stopping tablet replica
I20260927 18:39:11.680922 10633 raft_consensus.cc:2229] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:39:11.681334 10633 raft_consensus.cc:2258] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:11.681567 10634 ts_tablet_manager.cc:1905] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:11.682627 10633 ts_tablet_manager.cc:1905] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:11.687652 10635 tablet_replica.cc:330] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb: stopping tablet replica
I20260927 18:39:11.688030 10635 raft_consensus.cc:2229] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:11.688392 10635 raft_consensus.cc:2258] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:11.689508 10634 ts_tablet_manager.cc:1918] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:39:11.689591 10635 ts_tablet_manager.cc:1905] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:11.689771 10634 log.cc:1196] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/fde342f451b44f20b8e32edc45a6ecb8
I20260927 18:39:11.690171 10633 ts_tablet_manager.cc:1918] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:39:11.690318 10634 ts_tablet_manager.cc:1939] T fde342f451b44f20b8e32edc45a6ecb8 P 1ff04a7d37154373b757e1febc0b378a: Deleting consensus metadata
I20260927 18:39:11.690394 10633 log.cc:1196] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/fde342f451b44f20b8e32edc45a6ecb8
I20260927 18:39:11.690901 10633 ts_tablet_manager.cc:1939] T fde342f451b44f20b8e32edc45a6ecb8 P e6dccb7c09f34d348fa44bee9260a5e2: Deleting consensus metadata
I20260927 18:39:11.691527 10043 catalog_manager.cc:4841] TS 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267): tablet fde342f451b44f20b8e32edc45a6ecb8 (table test-table [id=4699507213f94e0d8fb49d41695a7199]) successfully deleted
I20260927 18:39:11.692020 10043 catalog_manager.cc:4841] TS e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66:40781): tablet fde342f451b44f20b8e32edc45a6ecb8 (table test-table [id=4699507213f94e0d8fb49d41695a7199]) successfully deleted
I20260927 18:39:11.696543 10635 ts_tablet_manager.cc:1918] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:39:11.696715 10635 log.cc:1196] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/fde342f451b44f20b8e32edc45a6ecb8
I20260927 18:39:11.697225 10635 ts_tablet_manager.cc:1939] T fde342f451b44f20b8e32edc45a6ecb8 P 76a51c8b6eb343539951dcf9791675cb: Deleting consensus metadata
I20260927 18:39:11.698429 10045 catalog_manager.cc:4841] TS 76a51c8b6eb343539951dcf9791675cb (127.2.95.67:33177): tablet fde342f451b44f20b8e32edc45a6ecb8 (table test-table [id=4699507213f94e0d8fb49d41695a7199]) successfully deleted
Sep 27 18:39:11 dist-test-slave-9jll krb5kdc[10013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 18:39:11 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:11 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:39:11.741816 10604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:11.729503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.2.95.126:37211)
0927 18:39:11.729736 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:11.729744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:11.730175 (+ 431us) server_negotiation.cc:407] Connection header received
0927 18:39:11.730424 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:11.730429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:11.730511 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:11.730608 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:11.732323 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.733165 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.735146 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.735597 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.735709 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.741481 (+ 5772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50374: BlockingRecv error: recv got EOF from 127.0.0.1:50374 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 18:39:11 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM
I20260927 18:39:11.759621 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.744807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50390 (local address 127.2.95.126:37211)
0927 18:39:11.745136 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:11.745143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:11.745361 (+ 218us) server_negotiation.cc:407] Connection header received
0927 18:39:11.745563 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:11.745568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:11.745631 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:11.745778 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:11.747033 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.747764 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.749122 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.749538 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.749657 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.754052 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:11.754058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:11.754061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:11.754099 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:11.756572 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:11.757540 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:11.757546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:11.757549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:11.757772 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:11.758187 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:11.758191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:11.758193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:11.758470 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:11.758722 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:11.759059 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 18:39:11.759289 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:39:11.763523 10057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:39:11.764132 10057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:39:11.770788 10194 tablet_service.cc:1467] Processing CreateTablet for tablet 1b99bdebe72d4cfb9901cbdb16693595 (DEFAULT_TABLE table=test-table [id=a6c67262157c45bf9f3fb8e092e3c2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:11.771116 10473 tablet_service.cc:1467] Processing CreateTablet for tablet 1b99bdebe72d4cfb9901cbdb16693595 (DEFAULT_TABLE table=test-table [id=a6c67262157c45bf9f3fb8e092e3c2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:11.771292 10194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b99bdebe72d4cfb9901cbdb16693595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:11.771359 10334 tablet_service.cc:1467] Processing CreateTablet for tablet 1b99bdebe72d4cfb9901cbdb16693595 (DEFAULT_TABLE table=test-table [id=a6c67262157c45bf9f3fb8e092e3c2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:11.771544 10473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b99bdebe72d4cfb9901cbdb16693595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:11.771720 10334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b99bdebe72d4cfb9901cbdb16693595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:11.778107 10567 tablet_bootstrap.cc:492] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a: Bootstrap starting.
I20260927 18:39:11.778438 10568 tablet_bootstrap.cc:492] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: Bootstrap starting.
I20260927 18:39:11.779636 10568 tablet_bootstrap.cc:654] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:11.779702 10567 tablet_bootstrap.cc:654] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:11.781553 10570 tablet_bootstrap.cc:492] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb: Bootstrap starting.
I20260927 18:39:11.782122 10568 tablet_bootstrap.cc:492] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: No bootstrap required, opened a new log
I20260927 18:39:11.782233 10568 ts_tablet_manager.cc:1397] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 18:39:11.782843 10568 raft_consensus.cc:348] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.783069 10568 raft_consensus.cc:374] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:11.783144 10568 raft_consensus.cc:729] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dccb7c09f34d348fa44bee9260a5e2, State: Initialized, Role: FOLLOWER
I20260927 18:39:11.783334 10570 tablet_bootstrap.cc:654] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:11.783373 10568 consensus_queue.cc:260] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.783849 10568 ts_tablet_manager.cc:1428] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 18:39:11.785039 10567 tablet_bootstrap.cc:492] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a: No bootstrap required, opened a new log
I20260927 18:39:11.785200 10567 ts_tablet_manager.cc:1397] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260927 18:39:11.786034 10567 raft_consensus.cc:348] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.786329 10567 raft_consensus.cc:374] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:11.786442 10567 raft_consensus.cc:729] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff04a7d37154373b757e1febc0b378a, State: Initialized, Role: FOLLOWER
I20260927 18:39:11.786672 10567 consensus_queue.cc:260] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.787317 10567 ts_tablet_manager.cc:1428] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:39:11.795885 10570 tablet_bootstrap.cc:492] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb: No bootstrap required, opened a new log
I20260927 18:39:11.796072 10570 ts_tablet_manager.cc:1397] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.002s
I20260927 18:39:11.796890 10570 raft_consensus.cc:348] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.797183 10570 raft_consensus.cc:374] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:11.797284 10570 raft_consensus.cc:729] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a51c8b6eb343539951dcf9791675cb, State: Initialized, Role: FOLLOWER
I20260927 18:39:11.797519 10570 consensus_queue.cc:260] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.798139 10570 ts_tablet_manager.cc:1428] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:39:11.802815 10575 raft_consensus.cc:482] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:39:11.803056 10575 raft_consensus.cc:504] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.803678 10575 leader_election.cc:288] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267), e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66:40781)
I20260927 18:39:11.804283 10354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b99bdebe72d4cfb9901cbdb16693595" candidate_uuid: "76a51c8b6eb343539951dcf9791675cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" is_pre_election: true
I20260927 18:39:11.804292 10215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b99bdebe72d4cfb9901cbdb16693595" candidate_uuid: "76a51c8b6eb343539951dcf9791675cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff04a7d37154373b757e1febc0b378a" is_pre_election: true
I20260927 18:39:11.804514 10354 raft_consensus.cc:2454] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76a51c8b6eb343539951dcf9791675cb in term 0.
I20260927 18:39:11.804584 10215 raft_consensus.cc:2454] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76a51c8b6eb343539951dcf9791675cb in term 0.
I20260927 18:39:11.805003 10425 leader_election.cc:302] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff04a7d37154373b757e1febc0b378a, 76a51c8b6eb343539951dcf9791675cb; no voters:
I20260927 18:39:11.805313 10575 raft_consensus.cc:2781] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:39:11.805433 10575 raft_consensus.cc:482] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:39:11.805522 10575 raft_consensus.cc:3037] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:11.807864 10575 raft_consensus.cc:504] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.808425 10575 leader_election.cc:288] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [CANDIDATE]: Term 1 election: Requested vote from peers 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267), e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66:40781)
I20260927 18:39:11.808924 10215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b99bdebe72d4cfb9901cbdb16693595" candidate_uuid: "76a51c8b6eb343539951dcf9791675cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff04a7d37154373b757e1febc0b378a"
I20260927 18:39:11.809001 10354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b99bdebe72d4cfb9901cbdb16693595" candidate_uuid: "76a51c8b6eb343539951dcf9791675cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dccb7c09f34d348fa44bee9260a5e2"
I20260927 18:39:11.809134 10215 raft_consensus.cc:3037] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:11.809257 10354 raft_consensus.cc:3037] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:11.811527 10215 raft_consensus.cc:2454] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a51c8b6eb343539951dcf9791675cb in term 1.
I20260927 18:39:11.812031 10425 leader_election.cc:302] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff04a7d37154373b757e1febc0b378a, 76a51c8b6eb343539951dcf9791675cb; no voters:
I20260927 18:39:11.812278 10354 raft_consensus.cc:2454] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76a51c8b6eb343539951dcf9791675cb in term 1.
I20260927 18:39:11.812337 10575 raft_consensus.cc:2781] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:11.812536 10575 raft_consensus.cc:686] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 1 LEADER]: Becoming Leader. State: Replica: 76a51c8b6eb343539951dcf9791675cb, State: Running, Role: LEADER
I20260927 18:39:11.812884 10575 consensus_queue.cc:237] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.815501 10057 catalog_manager.cc:5495] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 76a51c8b6eb343539951dcf9791675cb (127.2.95.67). New cstate: current_term: 1 leader_uuid: "76a51c8b6eb343539951dcf9791675cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } health_report { overall_health: HEALTHY } } }
I20260927 18:39:11.830473 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:11.824918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.2.95.67:33177)
0927 18:39:11.825148 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:11.825154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:11.825268 (+ 114us) server_negotiation.cc:407] Connection header received
0927 18:39:11.825461 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:11.825466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:11.825546 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:11.825634 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:39:11.826018 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.826845 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.828266 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:11.828791 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:11.828915 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:11.829407 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:39:11.829529 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:39:11.829612 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:11.830132 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 18:39:11.830220 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 18:39:11.834081 10215 raft_consensus.cc:1261] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Refusing update from remote peer 76a51c8b6eb343539951dcf9791675cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:11.834095 10354 raft_consensus.cc:1261] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Refusing update from remote peer 76a51c8b6eb343539951dcf9791675cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:11.834713 10625 consensus_queue.cc:1035] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:11.835333 10575 consensus_queue.cc:1035] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:11.850998 10057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:11.851296 10057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:11.856254 10057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6: Sending DeleteTablet for 3 replicas of tablet 1b99bdebe72d4cfb9901cbdb16693595
I20260927 18:39:11.857517 10334 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b99bdebe72d4cfb9901cbdb16693595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:11 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:41444
I20260927 18:39:11.857537 10194 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b99bdebe72d4cfb9901cbdb16693595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:11 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:47604
I20260927 18:39:11.857867 10633 tablet_replica.cc:330] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: stopping tablet replica
I20260927 18:39:11.857913 10634 tablet_replica.cc:330] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a: stopping tablet replica
I20260927 18:39:11.858057 10633 raft_consensus.cc:2229] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:11.858114 10634 raft_consensus.cc:2229] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:11.858197 10633 pending_rounds.cc:62] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: Trying to abort 2 pending ops.
I20260927 18:39:11.858328 10634 raft_consensus.cc:2258] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:11.858301 10633 pending_rounds.cc:68] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334028705105276928 op_type: WRITE_OP write_request { tablet_id: "1b99bdebe72d4cfb9901cbdb16693595" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\373\307\345\325\006\"8\n\ntest-admin\022*\n a6c67262157c45bf9f3fb8e092e3c2b9\020\001\030\001 \001(\001" signature: "<redacted>"":L\211\337\330l\334\252\263\022\3027X\036:)|\350\026\005\313\202\216~\371!a\250Z\203\343\247\033^B\360\262\375w5\357\370J\032\346A\363\340\305\325\034\327\245\t\023n\253~\014\324\'%\332I" signing_key_seq_num: 0 } } request_id { client_id: "bbb6cbba302e41f785a35e8f37105fec" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260927 18:39:11.859004 10633 pending_rounds.cc:68] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334028705027866624 op_type: NO_OP noop_request { }
I20260927 18:39:11.859148 10633 raft_consensus.cc:2866] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260927 18:39:11.859251 10633 raft_consensus.cc:2258] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:11.860013 10634 ts_tablet_manager.cc:1905] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:11.859933 10056 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:39:11.860785 10056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:39:11.864114 10633 ts_tablet_manager.cc:1905] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:11.865832 10473 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b99bdebe72d4cfb9901cbdb16693595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:11 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:35816
I20260927 18:39:11.866159 10635 tablet_replica.cc:330] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb: stopping tablet replica
I20260927 18:39:11.866377 10635 raft_consensus.cc:2229] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:39:11.866940 10635 raft_consensus.cc:2258] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:11.868286 10634 ts_tablet_manager.cc:1918] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:11.868575 10634 log.cc:1196] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/1b99bdebe72d4cfb9901cbdb16693595
I20260927 18:39:11.868922 10635 ts_tablet_manager.cc:1905] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:11.869215 10634 ts_tablet_manager.cc:1939] T 1b99bdebe72d4cfb9901cbdb16693595 P 1ff04a7d37154373b757e1febc0b378a: Deleting consensus metadata
I20260927 18:39:11.870395 10043 catalog_manager.cc:4841] TS 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267): tablet 1b99bdebe72d4cfb9901cbdb16693595 (table test-table [id=a6c67262157c45bf9f3fb8e092e3c2b9]) successfully deleted
I20260927 18:39:11.870824 10194 tablet_service.cc:1467] Processing CreateTablet for tablet 4a2c2269ba21430bbf07d029cff2914f (DEFAULT_TABLE table=test-table [id=69a6888633a24e8288b0cef172481a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:11.871235 10194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a2c2269ba21430bbf07d029cff2914f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:11.871870 10473 tablet_service.cc:1467] Processing CreateTablet for tablet 4a2c2269ba21430bbf07d029cff2914f (DEFAULT_TABLE table=test-table [id=69a6888633a24e8288b0cef172481a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:11.872464 10473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a2c2269ba21430bbf07d029cff2914f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:11.878649 10635 ts_tablet_manager.cc:1918] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:11.878789 10570 tablet_bootstrap.cc:492] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb: Bootstrap starting.
I20260927 18:39:11.878841 10635 log.cc:1196] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/1b99bdebe72d4cfb9901cbdb16693595
I20260927 18:39:11.878665 10633 ts_tablet_manager.cc:1918] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:11.879199 10334 tablet_service.cc:1467] Processing CreateTablet for tablet 4a2c2269ba21430bbf07d029cff2914f (DEFAULT_TABLE table=test-table [id=69a6888633a24e8288b0cef172481a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:11.879388 10633 log.cc:1196] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/1b99bdebe72d4cfb9901cbdb16693595
I20260927 18:39:11.879455 10635 ts_tablet_manager.cc:1939] T 1b99bdebe72d4cfb9901cbdb16693595 P 76a51c8b6eb343539951dcf9791675cb: Deleting consensus metadata
I20260927 18:39:11.879626 10334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a2c2269ba21430bbf07d029cff2914f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:11.879776 10633 ts_tablet_manager.cc:1939] T 1b99bdebe72d4cfb9901cbdb16693595 P e6dccb7c09f34d348fa44bee9260a5e2: Deleting consensus metadata
I20260927 18:39:11.880522 10570 tablet_bootstrap.cc:654] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:11.880882 10043 catalog_manager.cc:4841] TS e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66:40781): tablet 1b99bdebe72d4cfb9901cbdb16693595 (table test-table [id=a6c67262157c45bf9f3fb8e092e3c2b9]) successfully deleted
I20260927 18:39:11.882436 10045 catalog_manager.cc:4841] TS 76a51c8b6eb343539951dcf9791675cb (127.2.95.67:33177): tablet 1b99bdebe72d4cfb9901cbdb16693595 (table test-table [id=a6c67262157c45bf9f3fb8e092e3c2b9]) successfully deleted
I20260927 18:39:11.885715 10567 tablet_bootstrap.cc:492] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a: Bootstrap starting.
I20260927 18:39:11.886327 10568 tablet_bootstrap.cc:492] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2: Bootstrap starting.
I20260927 18:39:11.887274 10570 tablet_bootstrap.cc:492] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb: No bootstrap required, opened a new log
I20260927 18:39:11.887434 10567 tablet_bootstrap.cc:654] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:11.887444 10570 ts_tablet_manager.cc:1397] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260927 18:39:11.887923 10568 tablet_bootstrap.cc:654] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:11.888911 10570 raft_consensus.cc:348] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.889206 10570 raft_consensus.cc:374] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:11.889290 10570 raft_consensus.cc:729] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a51c8b6eb343539951dcf9791675cb, State: Initialized, Role: FOLLOWER
I20260927 18:39:11.889544 10570 consensus_queue.cc:260] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.890257 10570 ts_tablet_manager.cc:1428] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 18:39:11.896693 10568 tablet_bootstrap.cc:492] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2: No bootstrap required, opened a new log
I20260927 18:39:11.896898 10568 ts_tablet_manager.cc:1397] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260927 18:39:11.897209 10567 tablet_bootstrap.cc:492] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a: No bootstrap required, opened a new log
I20260927 18:39:11.897375 10567 ts_tablet_manager.cc:1397] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260927 18:39:11.897895 10568 raft_consensus.cc:348] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.898161 10568 raft_consensus.cc:374] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:11.898088 10567 raft_consensus.cc:348] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.898244 10568 raft_consensus.cc:729] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dccb7c09f34d348fa44bee9260a5e2, State: Initialized, Role: FOLLOWER
I20260927 18:39:11.898332 10567 raft_consensus.cc:374] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:11.898454 10567 raft_consensus.cc:729] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff04a7d37154373b757e1febc0b378a, State: Initialized, Role: FOLLOWER
I20260927 18:39:11.898492 10568 consensus_queue.cc:260] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.898736 10567 consensus_queue.cc:260] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:11.899060 10568 ts_tablet_manager.cc:1428] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 18:39:11.899768 10567 ts_tablet_manager.cc:1428] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 18:39:12.289044 10573 raft_consensus.cc:482] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:39:12.289256 10573 raft_consensus.cc:504] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:12.290859 10573 leader_election.cc:288] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66:40781), 76a51c8b6eb343539951dcf9791675cb (127.2.95.67:33177)
I20260927 18:39:12.309154 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.291354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:45171 (local address 127.2.95.66:40781)
0927 18:39:12.303259 (+ 11905us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:12.303266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:12.303284 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:39:12.303345 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:12.303350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:12.303423 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:12.303516 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:12.304248 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.305227 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.307126 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.308086 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.308224 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:12.308335 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:12.308681 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 18:39:12.308838 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11781,"thread_start_us":11690,"threads_started":1}
I20260927 18:39:12.309240 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.291091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:40781 (local address 127.2.95.65:45171)
0927 18:39:12.302631 (+ 11540us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:12.302659 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:12.302854 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:12.303589 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:12.303595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:12.303635 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:12.304064 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:12.304082 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.305346 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.305351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:12.306958 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:12.306973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.308282 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.308289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:12.308489 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:12.308530 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:39:12.308644 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 18:39:12.308796 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11345,"spinlock_wait_cycles":1280,"thread_start_us":12518,"threads_started":1}
I20260927 18:39:12.310105 10354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2c2269ba21430bbf07d029cff2914f" candidate_uuid: "1ff04a7d37154373b757e1febc0b378a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" is_pre_election: true
I20260927 18:39:12.310530 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.291266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:33177 (local address 127.2.95.65:47355)
0927 18:39:12.302659 (+ 11393us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:12.302673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:12.302853 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:12.303432 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:12.303437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:12.303465 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:12.303744 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:12.303760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.305217 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.305223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:12.307470 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:12.307488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.309600 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.309608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:12.309822 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:12.309875 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 18:39:12.309973 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:39:12.310128 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11301,"mutex_wait_us":1,"spinlock_wait_cycles":17280,"thread_start_us":11010,"threads_started":1}
I20260927 18:39:12.310580 10354 raft_consensus.cc:2454] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ff04a7d37154373b757e1febc0b378a in term 0.
I20260927 18:39:12.311311 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.291685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:47355 (local address 127.2.95.67:33177)
0927 18:39:12.291887 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:12.291893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:12.302811 (+ 10918us) server_negotiation.cc:407] Connection header received
0927 18:39:12.303069 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:12.303075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:12.303158 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:12.303222 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:12.303939 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.305039 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.307659 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.309098 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.309241 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:12.309370 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:12.310043 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 18:39:12.310186 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:39:12.311447 10146 leader_election.cc:302] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff04a7d37154373b757e1febc0b378a, e6dccb7c09f34d348fa44bee9260a5e2; no voters:
I20260927 18:39:12.311911 10573 raft_consensus.cc:2781] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:39:12.311894 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2c2269ba21430bbf07d029cff2914f" candidate_uuid: "1ff04a7d37154373b757e1febc0b378a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a51c8b6eb343539951dcf9791675cb" is_pre_election: true
I20260927 18:39:12.312062 10573 raft_consensus.cc:482] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:39:12.312150 10573 raft_consensus.cc:3037] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:12.312160 10492 raft_consensus.cc:2454] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ff04a7d37154373b757e1febc0b378a in term 0.
I20260927 18:39:12.315382 10573 raft_consensus.cc:504] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:12.316202 10573 leader_election.cc:288] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [CANDIDATE]: Term 1 election: Requested vote from peers e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66:40781), 76a51c8b6eb343539951dcf9791675cb (127.2.95.67:33177)
I20260927 18:39:12.316779 10492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2c2269ba21430bbf07d029cff2914f" candidate_uuid: "1ff04a7d37154373b757e1febc0b378a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a51c8b6eb343539951dcf9791675cb"
I20260927 18:39:12.316792 10354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2c2269ba21430bbf07d029cff2914f" candidate_uuid: "1ff04a7d37154373b757e1febc0b378a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6dccb7c09f34d348fa44bee9260a5e2"
I20260927 18:39:12.316996 10492 raft_consensus.cc:3037] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:12.317004 10354 raft_consensus.cc:3037] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:12.319339 10492 raft_consensus.cc:2454] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff04a7d37154373b757e1febc0b378a in term 1.
I20260927 18:39:12.319870 10354 raft_consensus.cc:2454] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff04a7d37154373b757e1febc0b378a in term 1.
I20260927 18:39:12.319893 10148 leader_election.cc:302] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff04a7d37154373b757e1febc0b378a, 76a51c8b6eb343539951dcf9791675cb; no voters:
I20260927 18:39:12.320257 10573 raft_consensus.cc:2781] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:12.320717 10573 raft_consensus.cc:686] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 1 LEADER]: Becoming Leader. State: Replica: 1ff04a7d37154373b757e1febc0b378a, State: Running, Role: LEADER
I20260927 18:39:12.321431 10573 consensus_queue.cc:237] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } }
I20260927 18:39:12.325896 10057 catalog_manager.cc:5495] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ff04a7d37154373b757e1febc0b378a (127.2.95.65). New cstate: current_term: 1 leader_uuid: "1ff04a7d37154373b757e1febc0b378a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff04a7d37154373b757e1febc0b378a" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 } health_report { overall_health: UNKNOWN } } }
I20260927 18:39:12.355553 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.349060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.2.95.65:33267)
0927 18:39:12.349817 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:12.349821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:12.350087 (+ 266us) server_negotiation.cc:407] Connection header received
0927 18:39:12.350380 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:12.350384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:12.350459 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:12.350647 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:39:12.351102 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.351874 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.353226 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.353655 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.353801 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:12.354249 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:39:12.354896 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:39:12.355025 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:12.355145 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 18:39:12.355231 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":513,"threads_started":1}
I20260927 18:39:12.372829 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.366337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:46863 (local address 127.2.95.67:33177)
0927 18:39:12.366579 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:12.366589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:12.367203 (+ 614us) server_negotiation.cc:407] Connection header received
0927 18:39:12.367373 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:12.367378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:12.367495 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:12.367603 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:39:12.368044 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.369157 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.370656 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.371309 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.371437 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:12.372007 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:39:12.372156 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:39:12.372244 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:12.372407 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 18:39:12.372499 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 18:39:12.372948 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.366336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:33177 (local address 127.2.95.65:46863)
0927 18:39:12.367048 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:12.367113 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:12.367246 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:12.367615 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:12.367618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:12.367627 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:39:12.367914 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:12.367922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.369317 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.369322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:12.370514 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:12.370528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.371572 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.371578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:12.371712 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:12.371777 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:39:12.372275 (+ 498us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:39:12.372286 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:39:12.372364 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 18:39:12.372477 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":339,"threads_started":1}
I20260927 18:39:12.381829 10492 raft_consensus.cc:1261] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Refusing update from remote peer 1ff04a7d37154373b757e1febc0b378a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:12.381799 10354 raft_consensus.cc:1261] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Refusing update from remote peer 1ff04a7d37154373b757e1febc0b378a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:12.382490 10661 consensus_queue.cc:1035] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6dccb7c09f34d348fa44bee9260a5e2" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 40781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:12.383154 10573 consensus_queue.cc:1035] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a51c8b6eb343539951dcf9791675cb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:12.418692 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.411801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:33267 (local address 127.2.95.67:35211)
0927 18:39:12.412332 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:12.412361 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:12.412485 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:12.413190 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:12.413194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:12.413208 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 18:39:12.413495 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:12.413508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.414961 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.414966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:12.416506 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:12.416522 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.417537 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.417544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:12.417747 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:12.417799 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 18:39:12.418193 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 18:39:12.418205 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 18:39:12.418280 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 18:39:12.418422 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":1208,"threads_started":1}
I20260927 18:39:12.418684 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.412674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:35211 (local address 127.2.95.65:33267)
0927 18:39:12.412882 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:12.412889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:12.412909 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:39:12.412979 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:12.412985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:12.413073 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:12.413171 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:39:12.413769 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.414753 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.416697 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.417305 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.417420 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:12.417952 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:39:12.418062 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:39:12.418143 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:12.418330 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 18:39:12.418420 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:39:12.459769 10057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:12.460089 10057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:12.463238 10057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d7754abcb1a40e192502fcd881c54f6: Sending DeleteTablet for 3 replicas of tablet 4a2c2269ba21430bbf07d029cff2914f
I20260927 18:39:12.464190 10194 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a2c2269ba21430bbf07d029cff2914f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:12 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:47604
I20260927 18:39:12.464226 10473 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a2c2269ba21430bbf07d029cff2914f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:12 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:35816
I20260927 18:39:12.464389 10334 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a2c2269ba21430bbf07d029cff2914f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:12 UTC) from {username='oryx', principal='oryx/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:41444
I20260927 18:39:12.466173 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 10129
I20260927 18:39:12.474891 10683 tablet_replica.cc:330] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2: stopping tablet replica
I20260927 18:39:12.475721 10683 raft_consensus.cc:2229] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:12.476344 10683 raft_consensus.cc:2258] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:12.478547 10683 ts_tablet_manager.cc:1905] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:12.491286 10683 ts_tablet_manager.cc:1918] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:39:12.496389 10683 log.cc:1196] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/4a2c2269ba21430bbf07d029cff2914f
I20260927 18:39:12.497332 10683 ts_tablet_manager.cc:1939] T 4a2c2269ba21430bbf07d029cff2914f P e6dccb7c09f34d348fa44bee9260a5e2: Deleting consensus metadata
I20260927 18:39:12.499957 10043 catalog_manager.cc:4841] TS e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66:40781): tablet 4a2c2269ba21430bbf07d029cff2914f (table test-table [id=69a6888633a24e8288b0cef172481a31]) successfully deleted
I20260927 18:39:12.502116 10681 tablet_replica.cc:330] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a: stopping tablet replica
I20260927 18:39:12.502378 10681 raft_consensus.cc:2229] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:39:12.503059 10681 raft_consensus.cc:2258] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:12.507266 10681 ts_tablet_manager.cc:1905] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:12.511655 10682 tablet_replica.cc:330] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb: stopping tablet replica
I20260927 18:39:12.511963 10682 raft_consensus.cc:2229] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:12.512272 10682 raft_consensus.cc:2258] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:12.513504 10682 ts_tablet_manager.cc:1905] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:12.516942 10681 ts_tablet_manager.cc:1918] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:39:12.517171 10681 log.cc:1196] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/4a2c2269ba21430bbf07d029cff2914f
I20260927 18:39:12.517746 10681 ts_tablet_manager.cc:1939] T 4a2c2269ba21430bbf07d029cff2914f P 1ff04a7d37154373b757e1febc0b378a: Deleting consensus metadata
I20260927 18:39:12.519138 10043 catalog_manager.cc:4841] TS 1ff04a7d37154373b757e1febc0b378a (127.2.95.65:33267): tablet 4a2c2269ba21430bbf07d029cff2914f (table test-table [id=69a6888633a24e8288b0cef172481a31]) successfully deleted
Sep 27 18:39:12 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.95.65@KRBTEST.COM
I20260927 18:39:12.521303 10682 ts_tablet_manager.cc:1918] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 18:39:12.521433 10682 log.cc:1196] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/4a2c2269ba21430bbf07d029cff2914f
I20260927 18:39:12.522881 10682 ts_tablet_manager.cc:1939] T 4a2c2269ba21430bbf07d029cff2914f P 76a51c8b6eb343539951dcf9791675cb: Deleting consensus metadata
I20260927 18:39:12.524120 10045 catalog_manager.cc:4841] TS 76a51c8b6eb343539951dcf9791675cb (127.2.95.67:33177): tablet 4a2c2269ba21430bbf07d029cff2914f (table test-table [id=69a6888633a24e8288b0cef172481a31]) successfully deleted
I20260927 18:39:12.532204 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.472599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47620 (local address 127.2.95.65:33267)
0927 18:39:12.475570 (+ 2971us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:12.475577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:12.494730 (+ 19153us) server_negotiation.cc:407] Connection header received
0927 18:39:12.495038 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:12.495045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:12.495140 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:12.495226 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:12.497526 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.498924 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.506078 (+ 7154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.506782 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.506951 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:12.522613 (+ 15662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:12.522632 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:12.522654 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:12.522715 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:12.527769 (+ 5054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:12.528798 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:12.528808 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:12.528812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:12.528996 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:12.529480 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:12.529485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:12.529489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:12.530036 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:12.530340 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:12.531552 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0927 18:39:12.531777 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2820}
I20260927 18:39:12.533361 10260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:12.658949 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 10129
I20260927 18:39:12.674592 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 10269
Sep 27 18:39:12 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.95.66@KRBTEST.COM
I20260927 18:39:12.690191 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.675462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.2.95.66:40781)
0927 18:39:12.675675 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:12.675681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:12.675699 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:39:12.675801 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:12.675805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:12.675867 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:12.675949 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:12.677253 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.678113 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.679355 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.679761 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.679863 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:12.684624 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:12.684640 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:12.684657 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:12.684706 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:12.687600 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:12.688366 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:12.688374 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:12.688376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:12.688568 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:12.688852 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:12.688855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:12.688858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:12.689219 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:12.689424 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:12.689724 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 18:39:12.689909 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 18:39:12.690843 10399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:12.813235 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 10269
I20260927 18:39:12.831403 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 10408
Sep 27 18:39:12 dist-test-slave-9jll krb5kdc[10013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.95.67@KRBTEST.COM
I20260927 18:39:12.842785 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.841879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:40781 (local address 127.2.95.67:37847)
0927 18:39:12.842374 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:12.842515 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.66:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":240,"threads_started":1}
W20260927 18:39:12.843181 10425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.95.66:40781: connect: Connection refused (error 111)
I20260927 18:39:12.844746 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.844415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:40781 (local address 127.2.95.67:59975)
0927 18:39:12.844554 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:12.844626 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.66:40781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 18:39:12.844944 10425 consensus_peers.cc:487] T b6601952fb43472c929bbdc47101b37c P 76a51c8b6eb343539951dcf9791675cb -> Peer e6dccb7c09f34d348fa44bee9260a5e2 (127.2.95.66:40781): Couldn't send request to peer e6dccb7c09f34d348fa44bee9260a5e2. Status: Network error: Client connection negotiation failed: client connection to 127.2.95.66:40781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 18:39:12.847414 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.832269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.2.95.67:33177)
0927 18:39:12.832457 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:12.832463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:12.832481 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:39:12.832531 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:12.832534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:12.832631 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:12.832712 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:12.834032 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.834742 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.835943 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:12.836397 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:12.836517 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:12.841015 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:12.841037 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:12.841059 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:12.841107 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:12.844837 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:12.845551 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:12.845558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:12.845561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:12.845790 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:12.846149 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:12.846153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:12.846155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:12.846569 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:12.846807 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:12.847015 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 18:39:12.847198 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:39:12.848043 10538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:12.990908 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 10408
I20260927 18:39:12.991039 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:12.990458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:33267 (local address 127.2.95.67:36505)
0927 18:39:12.990643 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:12.990760 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:33267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 18:39:13.010936 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 10026
I20260927 18:39:13.012135 10098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:13.126587 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 10026
2026-09-27T18:39:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5992 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:39:13 dist-test-slave-9jll krb5kdc[10693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:39:13 dist-test-slave-9jll krb5kdc[10693](info): set up 2 sockets
Sep 27 18:39:13 dist-test-slave-9jll krb5kdc[10693](info): commencing operation
krb5kdc: starting...
W20260927 18:39:14.154563 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:39:14 dist-test-slave-9jll krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:39:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:39:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:39:14.382908 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33881
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:45765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:33881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:14.710750 10706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:14.711090 10706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:14.711222 10706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:14.718724 10706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:39:14.718863 10706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:14.718942 10706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:14.719023 10706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:39:14.719082 10706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:39:14.729161 10706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:45765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:33881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:14.730537 10706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:14.731860 10706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:14.744195 10711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:14.744161 10712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:14.744131 10714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:14.744903 10706 server_base.cc:1034] running on GCE node
I20260927 18:39:14.745754 10706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:14.747236 10706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:14.748485 10706 hybrid_clock.cc:648] HybridClock initialized: now 1790534354748426 us; error 45 us; skew 500 ppm
I20260927 18:39:14.748893 10706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T18:39:14Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1707 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:39:14 dist-test-slave-9jll krb5kdc[10720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:39:14 dist-test-slave-9jll krb5kdc[10720](info): set up 2 sockets
Sep 27 18:39:14 dist-test-slave-9jll krb5kdc[10720](info): commencing operation
krb5kdc: starting...
W20260927 18:39:15.866523 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:39:16 dist-test-slave-9jll krb5kdc[10720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:39:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:39:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:39:16.090399 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:38773
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:38773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:16.418771 10733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:16.419096 10733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:16.419211 10733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:16.426946 10733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:39:16.427074 10733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:16.427122 10733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:16.427201 10733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:39:16.427251 10733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:39:16.437693 10733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:38773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:38773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:16.439045 10733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:16.440332 10733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:16.452351 10741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:16.452378 10739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:16.452468 10738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:16.453089 10733 server_base.cc:1034] running on GCE node
I20260927 18:39:16.453994 10733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:16.455483 10733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:16.456713 10733 hybrid_clock.cc:648] HybridClock initialized: now 1790534356456657 us; error 49 us; skew 500 ppm
I20260927 18:39:16.457129 10733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:16 dist-test-slave-9jll krb5kdc[10720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534356, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:16.460245 10733 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:39:16.461861 10733 webserver.cc:460] Webserver started at http://127.2.95.126:36003/ using document root <none> and password file <none>
I20260927 18:39:16.462594 10733 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:16.462704 10733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:16.462994 10733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:16.466102 10733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "8c143fbbb1a4464899efb3c015f717db"
format_stamp: "Formatted at 2026-09-27 18:39:16 on dist-test-slave-9jll"
server_key: "1e8ed3ab5f3243bac3f0f76fead6b4d0"
server_key_iv: "92dee1e1e24b02e42183f755d02fb311"
server_key_version: "encryptionkey@0"
I20260927 18:39:16.466871 10733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "8c143fbbb1a4464899efb3c015f717db"
format_stamp: "Formatted at 2026-09-27 18:39:16 on dist-test-slave-9jll"
server_key: "1e8ed3ab5f3243bac3f0f76fead6b4d0"
server_key_iv: "92dee1e1e24b02e42183f755d02fb311"
server_key_version: "encryptionkey@0"
I20260927 18:39:16.472137 10733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:39:16.475878 10748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:16.477269 10733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:39:16.477433 10733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "8c143fbbb1a4464899efb3c015f717db"
format_stamp: "Formatted at 2026-09-27 18:39:16 on dist-test-slave-9jll"
server_key: "1e8ed3ab5f3243bac3f0f76fead6b4d0"
server_key_iv: "92dee1e1e24b02e42183f755d02fb311"
server_key_version: "encryptionkey@0"
I20260927 18:39:16.477614 10733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:16.518033 10733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:16.522584 10733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:16.522861 10733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:16.541612 10733 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:38773
I20260927 18:39:16.541703 10799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:38773 every 8 connection(s)
I20260927 18:39:16.543094 10733 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:39:16.544279 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 10733
I20260927 18:39:16.544519 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:39:16.545135 2429 external_mini_cluster.cc:1428] Setting key 34a4f98175186990e9dadd45c0fc9efa
I20260927 18:39:16.547546 10800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:39:16 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:39:16.561019 10800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db: Bootstrap starting.
I20260927 18:39:16.565052 10800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:16.566509 10800 log.cc:824] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:16.570173 10800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db: No bootstrap required, opened a new log
I20260927 18:39:16.571514 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:16.547140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.2.95.126:38773)
0927 18:39:16.548046 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:16.548061 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:16.548109 (+ 48us) server_negotiation.cc:407] Connection header received
0927 18:39:16.550406 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:16.550435 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:16.550998 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:16.551483 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:16.552992 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:16.555040 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:16.556460 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:16.557527 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:16.557795 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:16.562141 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:16.562153 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:16.562175 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:16.562227 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:16.566370 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:16.567458 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:16.567469 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:16.567477 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:16.567690 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:16.568131 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:16.568136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:16.568139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:16.568863 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:16.569156 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:16.569569 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 18:39:16.570087 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":397,"threads_started":1}
I20260927 18:39:16.578886 10800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c143fbbb1a4464899efb3c015f717db" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38773 } }
I20260927 18:39:16.579345 10800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:16.579463 10800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c143fbbb1a4464899efb3c015f717db, State: Initialized, Role: FOLLOWER
I20260927 18:39:16.580041 10800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c143fbbb1a4464899efb3c015f717db" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38773 } }
I20260927 18:39:16.580259 10800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:39:16.580386 10800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:39:16.580540 10800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:16.582860 10800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c143fbbb1a4464899efb3c015f717db" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38773 } }
I20260927 18:39:16.583371 10800 leader_election.cc:302] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c143fbbb1a4464899efb3c015f717db; no voters:
I20260927 18:39:16.583925 10800 leader_election.cc:288] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:39:16.584126 10805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:16.584743 10805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 1 LEADER]: Becoming Leader. State: Replica: 8c143fbbb1a4464899efb3c015f717db, State: Running, Role: LEADER
I20260927 18:39:16.585296 10805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c143fbbb1a4464899efb3c015f717db" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38773 } }
I20260927 18:39:16.585870 10800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:39:16.587795 10807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c143fbbb1a4464899efb3c015f717db. Latest consensus state: current_term: 1 leader_uuid: "8c143fbbb1a4464899efb3c015f717db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c143fbbb1a4464899efb3c015f717db" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38773 } } }
I20260927 18:39:16.587818 10806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c143fbbb1a4464899efb3c015f717db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c143fbbb1a4464899efb3c015f717db" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38773 } } }
I20260927 18:39:16.588349 10807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:16.588369 10806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:16.590968 10810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:39:16.595456 10810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:39:16.603464 10810 catalog_manager.cc:1347] Generated new cluster ID: 6346daac90e4464da609e94c10367004
I20260927 18:39:16.603595 10810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:39:16.613214 10810 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:39:16.614696 10810 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:39:16.621898 10810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db: Generated new TSK 0
I20260927 18:39:16.622861 10810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:39:16.684046 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:38773
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:17.022619 10826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:17.022969 10826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:17.023106 10826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:17.030932 10826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:17.031075 10826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:17.031230 10826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:39:17.041241 10826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:38773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:17.042630 10826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:17.044041 10826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:17.057482 10831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:17.057505 10834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:17.057461 10832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:17.058128 10826 server_base.cc:1034] running on GCE node
I20260927 18:39:17.058867 10826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:17.059919 10826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:17.061153 10826 hybrid_clock.cc:648] HybridClock initialized: now 1790534357061109 us; error 40 us; skew 500 ppm
I20260927 18:39:17.061532 10826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:17 dist-test-slave-9jll krb5kdc[10720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534357, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:17.064770 10826 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:39:17.066449 10826 webserver.cc:460] Webserver started at http://127.2.95.65:42235/ using document root <none> and password file <none>
I20260927 18:39:17.067131 10826 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:17.067250 10826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:17.067557 10826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:17.070691 10826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "2bc7181be9ed4cfda98375c7d75faa72"
format_stamp: "Formatted at 2026-09-27 18:39:17 on dist-test-slave-9jll"
server_key: "34b7be42cea4402a39086be54b9ce52f"
server_key_iv: "b6b9a1c608be1d0f88e8aea95658a6ac"
server_key_version: "encryptionkey@0"
I20260927 18:39:17.071491 10826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "2bc7181be9ed4cfda98375c7d75faa72"
format_stamp: "Formatted at 2026-09-27 18:39:17 on dist-test-slave-9jll"
server_key: "34b7be42cea4402a39086be54b9ce52f"
server_key_iv: "b6b9a1c608be1d0f88e8aea95658a6ac"
server_key_version: "encryptionkey@0"
I20260927 18:39:17.076920 10826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:39:17.080672 10841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:17.082227 10826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 18:39:17.082398 10826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "2bc7181be9ed4cfda98375c7d75faa72"
format_stamp: "Formatted at 2026-09-27 18:39:17 on dist-test-slave-9jll"
server_key: "34b7be42cea4402a39086be54b9ce52f"
server_key_iv: "b6b9a1c608be1d0f88e8aea95658a6ac"
server_key_version: "encryptionkey@0"
I20260927 18:39:17.082641 10826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:17.101842 10826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:17.106213 10826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:17.106503 10826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:17.107430 10826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:17.108795 10826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:17.108872 10826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:17.109030 10826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:17.109083 10826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:17.142855 10826 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:32915
I20260927 18:39:17.142900 10953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:32915 every 8 connection(s)
I20260927 18:39:17.144254 10826 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:39:17.149307 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 10826
I20260927 18:39:17.149521 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:39:17.150308 2429 external_mini_cluster.cc:1428] Setting key 1e9d9468e48e6a00132241cf61b6cf05
Sep 27 18:39:17 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534357, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:39:17.167207 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:17.146873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:49871 (local address 127.2.95.126:38773)
0927 18:39:17.147311 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:17.147318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:17.148423 (+ 1105us) server_negotiation.cc:407] Connection header received
0927 18:39:17.150861 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:17.150865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:17.150946 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:17.151073 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:17.153215 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:17.154039 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:17.156230 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:17.156653 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:17.156763 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:17.160876 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:17.160883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:17.160887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:17.160932 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:17.163646 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:17.164639 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:17.164644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:17.164647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:17.164805 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:17.165309 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:17.165313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:17.165315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:17.165615 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:17.165787 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:17.166659 (+ 872us) server_negotiation.cc:299] Negotiation successful
0927 18:39:17.166843 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":209,"threads_started":1}
I20260927 18:39:17.168497 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:17.147471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:49871)
0927 18:39:17.148249 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:17.148303 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:17.150531 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:17.151510 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:17.151523 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:17.152107 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:17.152898 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:17.153028 (+ 130us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:17.154247 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:17.154252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:17.156049 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:17.156065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:17.156887 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:17.156893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:17.157190 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:17.158004 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:17.158035 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:17.160594 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:17.163818 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:17.163829 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:17.163848 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:17.164486 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:17.164932 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:17.164937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:17.164941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:17.165185 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:17.165769 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:17.165778 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:17.166250 (+ 472us) client_negotiation.cc:769] Sending connection context
0927 18:39:17.166562 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 18:39:17.167154 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":284,"threads_started":1}
I20260927 18:39:17.170254 10954 heartbeater.cc:347] Connected to a master server at 127.2.95.126:38773
I20260927 18:39:17.170593 10954 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:17.171337 10954 heartbeater.cc:511] Master 127.2.95.126:38773 requested a full tablet report, sending...
I20260927 18:39:17.173889 10765 ts_manager.cc:194] Registered new tserver with Master: 2bc7181be9ed4cfda98375c7d75faa72 (127.2.95.65:32915)
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
I20260927 18:39:17.175643 10765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:49871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:39:17.205358 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:38773
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:17.529626 10960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:17.529994 10960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:17.530105 10960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:17.537516 10960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:17.537640 10960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:17.537808 10960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:39:17.547856 10960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:38773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:17.549171 10960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:17.550494 10960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:17.563663 10966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:17.563601 10965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:17.563637 10968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:17.564381 10960 server_base.cc:1034] running on GCE node
I20260927 18:39:17.565027 10960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:17.566118 10960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:17.567318 10960 hybrid_clock.cc:648] HybridClock initialized: now 1790534357567266 us; error 51 us; skew 500 ppm
I20260927 18:39:17.567632 10960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:17 dist-test-slave-9jll krb5kdc[10720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534357, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:17.570999 10960 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:39:17.572517 10960 webserver.cc:460] Webserver started at http://127.2.95.66:42455/ using document root <none> and password file <none>
I20260927 18:39:17.573210 10960 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:17.573308 10960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:17.573570 10960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:17.576566 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "a6d8cd8bb7204aedb789e7594433858d"
format_stamp: "Formatted at 2026-09-27 18:39:17 on dist-test-slave-9jll"
server_key: "c54239e6b0878c0ae60a874daf4dff6b"
server_key_iv: "d53d99795ab706b4ada977c6e30d774b"
server_key_version: "encryptionkey@0"
I20260927 18:39:17.577353 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "a6d8cd8bb7204aedb789e7594433858d"
format_stamp: "Formatted at 2026-09-27 18:39:17 on dist-test-slave-9jll"
server_key: "c54239e6b0878c0ae60a874daf4dff6b"
server_key_iv: "d53d99795ab706b4ada977c6e30d774b"
server_key_version: "encryptionkey@0"
I20260927 18:39:17.582794 10960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:39:17.586684 10975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:17.588115 10960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:39:17.588263 10960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "a6d8cd8bb7204aedb789e7594433858d"
format_stamp: "Formatted at 2026-09-27 18:39:17 on dist-test-slave-9jll"
server_key: "c54239e6b0878c0ae60a874daf4dff6b"
server_key_iv: "d53d99795ab706b4ada977c6e30d774b"
server_key_version: "encryptionkey@0"
I20260927 18:39:17.588461 10960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:17.633378 10960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:17.638155 10960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:17.638441 10960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:17.639542 10960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:17.641024 10960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:17.641103 10960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:17.641187 10960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:17.641229 10960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:17.672636 10960 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:43289
I20260927 18:39:17.672749 11087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:43289 every 8 connection(s)
I20260927 18:39:17.673981 10960 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:39:17.677136 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 10960
I20260927 18:39:17.677311 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:39:17.677809 2429 external_mini_cluster.cc:1428] Setting key ef6813cc9aada620cc20ad678567d541
Sep 27 18:39:17 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534357, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:39:17.695456 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:17.676763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:55417 (local address 127.2.95.126:38773)
0927 18:39:17.677289 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:17.677296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:17.678357 (+ 1061us) server_negotiation.cc:407] Connection header received
0927 18:39:17.680472 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:17.680477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:17.680561 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:17.680675 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:17.682360 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:17.683279 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:17.685061 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:17.685622 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:17.685773 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:17.689562 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:17.689568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:17.689572 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:17.689606 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:17.692289 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:17.693099 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:17.693103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:17.693106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:17.693230 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:17.693710 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:17.693717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:17.693721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:17.694015 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:17.694155 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:17.694851 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 18:39:17.695091 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":310,"threads_started":1}
I20260927 18:39:17.696584 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:17.677381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:55417)
0927 18:39:17.678168 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:17.678222 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:17.680196 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:17.680915 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:17.680927 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:17.681503 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:17.682189 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:17.682208 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:17.683468 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:17.683474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:17.684903 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:17.684924 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:17.685919 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:17.685928 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:17.686161 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:17.686906 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:17.686930 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:17.689315 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:17.692444 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:17.692454 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:17.692472 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:17.692969 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:17.693341 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:17.693346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:17.693350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:17.693552 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:17.694136 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:17.694144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:17.694499 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 18:39:17.694753 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 18:39:17.695316 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":312,"threads_started":1}
I20260927 18:39:17.698204 11088 heartbeater.cc:347] Connected to a master server at 127.2.95.126:38773
I20260927 18:39:17.698498 11088 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
I20260927 18:39:17.699223 11088 heartbeater.cc:511] Master 127.2.95.126:38773 requested a full tablet report, sending...
I20260927 18:39:17.700731 10765 ts_manager.cc:194] Registered new tserver with Master: a6d8cd8bb7204aedb789e7594433858d (127.2.95.66:43289)
I20260927 18:39:17.701627 10765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:55417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:39:17.731323 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:38773
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:18.059911 11094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:18.060225 11094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:18.060317 11094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:18.068073 11094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:18.068213 11094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:18.068328 11094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:39:18.078521 11094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:38773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:18.079774 11094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:18.081012 11094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:18.094884 11100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:18.094966 11102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:18.094945 11099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:18.095839 11094 server_base.cc:1034] running on GCE node
I20260927 18:39:18.096442 11094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:18.097434 11094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:18.098663 11094 hybrid_clock.cc:648] HybridClock initialized: now 1790534358098591 us; error 64 us; skew 500 ppm
I20260927 18:39:18.098981 11094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:18 dist-test-slave-9jll krb5kdc[10720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534358, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:18.102150 11094 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:39:18.103729 11094 webserver.cc:460] Webserver started at http://127.2.95.67:46047/ using document root <none> and password file <none>
I20260927 18:39:18.104411 11094 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:18.104523 11094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:18.104825 11094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:18.107841 11094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "50076f866b064a79a776e859c6642ccd"
format_stamp: "Formatted at 2026-09-27 18:39:18 on dist-test-slave-9jll"
server_key: "3fea02b5790e709ef940aa25568de467"
server_key_iv: "3b7e1b2da08ca5e878fd911e26293430"
server_key_version: "encryptionkey@0"
I20260927 18:39:18.108594 11094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "50076f866b064a79a776e859c6642ccd"
format_stamp: "Formatted at 2026-09-27 18:39:18 on dist-test-slave-9jll"
server_key: "3fea02b5790e709ef940aa25568de467"
server_key_iv: "3b7e1b2da08ca5e878fd911e26293430"
server_key_version: "encryptionkey@0"
I20260927 18:39:18.113925 11094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 18:39:18.117826 11109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:18.119355 11094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:39:18.119566 11094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "50076f866b064a79a776e859c6642ccd"
format_stamp: "Formatted at 2026-09-27 18:39:18 on dist-test-slave-9jll"
server_key: "3fea02b5790e709ef940aa25568de467"
server_key_iv: "3b7e1b2da08ca5e878fd911e26293430"
server_key_version: "encryptionkey@0"
I20260927 18:39:18.119771 11094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:18.144444 11094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:18.148888 11094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:18.149158 11094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:18.150333 11094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:18.151701 11094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:18.151895 11094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:18.152055 11094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:18.152124 11094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:18.179442 10954 heartbeater.cc:503] Master 127.2.95.126:38773 was elected leader, sending a full tablet report...
I20260927 18:39:18.183729 11094 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:41763
I20260927 18:39:18.183776 11221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:41763 every 8 connection(s)
I20260927 18:39:18.185043 11094 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:39:18.193521 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 11094
I20260927 18:39:18.193712 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:39:18.194144 2429 external_mini_cluster.cc:1428] Setting key 15c0289f53245ab4d36a800f7ca7ce4d
Sep 27 18:39:18 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534358, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:39:18.206357 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:18.187785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:46311 (local address 127.2.95.126:38773)
0927 18:39:18.187994 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:18.188001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:18.189189 (+ 1188us) server_negotiation.cc:407] Connection header received
0927 18:39:18.191307 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:18.191312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:18.191396 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:18.191488 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:18.193333 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:18.194183 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:18.196019 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:18.196567 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:18.196703 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:18.200728 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:18.200734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:18.200737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:18.200774 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:18.203340 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:18.204258 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:18.204263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:18.204265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:18.204386 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:18.204778 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:18.204782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:18.204784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:18.205064 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:18.205174 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:18.205955 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 18:39:18.206134 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:39:18.207449 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:18.188268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:46311)
0927 18:39:18.189016 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:18.189056 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:18.190959 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:18.191733 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:18.191746 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:18.192441 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:18.193101 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:18.193123 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:18.194363 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:18.194367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:18.195872 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:18.195887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:18.196852 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:18.196857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:18.197256 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:18.198081 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:18.198106 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:18.200435 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:18.203524 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:18.203533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:18.203550 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:18.204125 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:18.204490 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:18.204494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:18.204496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:18.204685 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:18.205191 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:18.205198 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:18.205546 (+ 348us) client_negotiation.cc:769] Sending connection context
0927 18:39:18.205857 (+ 311us) client_negotiation.cc:240] Negotiation successful
0927 18:39:18.206347 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":291,"threads_started":1}
I20260927 18:39:18.209033 11222 heartbeater.cc:347] Connected to a master server at 127.2.95.126:38773
I20260927 18:39:18.209352 11222 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:18.210108 11222 heartbeater.cc:511] Master 127.2.95.126:38773 requested a full tablet report, sending...
I20260927 18:39:18.211512 10764 ts_manager.cc:194] Registered new tserver with Master: 50076f866b064a79a776e859c6642ccd (127.2.95.67:41763)
I20260927 18:39:18.212399 10764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:46311
I20260927 18:39:18.222473 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab.
Entry for principal oryx/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab.
WARNING: no policy specified for oryx/127.2.95.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab.
Entry for principal oryx/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab.
WARNING: no policy specified for oryx/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab.
Entry for principal oryx/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab.
I20260927 18:39:18.377338 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 10826
Sep 27 18:39:18 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:39:18.396229 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:18.378439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38432 (local address 127.2.95.65:32915)
0927 18:39:18.379055 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:18.379062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:18.379093 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:39:18.379194 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:18.379200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:18.379419 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:18.379555 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:18.381007 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:18.382691 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:18.384571 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:18.385522 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:18.385729 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:18.390314 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:18.390328 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:18.390344 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:18.390392 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:18.393349 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:18.394136 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:18.394143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:18.394146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:18.394281 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:18.394660 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:18.394664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:18.394666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:18.395035 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:18.395312 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:18.395610 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 18:39:18.395866 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":284,"threads_started":1}
I20260927 18:39:18.397578 10949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:18.480655 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 10826
I20260927 18:39:18.494800 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 10960
Sep 27 18:39:18 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:39:18.511765 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:18.495668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.2.95.66:43289)
0927 18:39:18.496225 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:18.496233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:18.496258 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:39:18.496314 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:18.496319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:18.496539 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:18.496688 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:18.498144 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:18.499239 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:18.500574 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:18.501220 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:18.501319 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:18.505736 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:18.505749 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:18.505765 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:18.505805 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:18.508597 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:18.509362 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:18.509369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:18.509372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:18.509544 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:18.510059 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:18.510064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:18.510067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:18.510442 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:18.510679 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:18.511136 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 18:39:18.511406 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":316,"threads_started":1}
I20260927 18:39:18.513024 11083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:18.600072 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 10960
I20260927 18:39:18.612660 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 11094
Sep 27 18:39:18 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:39:18.629438 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:18.613740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.2.95.67:41763)
0927 18:39:18.614452 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:18.614459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:18.614490 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:39:18.614548 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:18.614554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:18.614783 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:18.614952 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:18.616334 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:18.617392 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:18.618732 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:18.619285 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:18.619399 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:18.623804 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:18.623816 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:18.623831 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:18.623878 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:18.626643 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:18.627378 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:18.627386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:18.627388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:18.627522 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:18.627886 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:18.627890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:18.627893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:18.628249 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:18.628515 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:18.628837 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 18:39:18.629105 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":394,"threads_started":1}
I20260927 18:39:18.630784 11217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:18.714497 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 11094
I20260927 18:39:18.727331 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 10733
I20260927 18:39:18.728490 10795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:18.830183 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 10733
I20260927 18:39:18.848227 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:38773
--webserver_interface=127.2.95.126
--webserver_port=36003
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:38773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:19.178547 11239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:19.178850 11239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:19.178989 11239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:19.186362 11239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:39:19.186481 11239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:19.186553 11239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:19.186628 11239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:39:19.186687 11239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:39:19.197070 11239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:38773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:38773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=36003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:19.198459 11239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:19.199765 11239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:19.212306 11245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:19.212360 11247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:19.212340 11244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:19.213096 11239 server_base.cc:1034] running on GCE node
I20260927 18:39:19.213927 11239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:19.215366 11239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:19.216595 11239 hybrid_clock.cc:648] HybridClock initialized: now 1790534359216520 us; error 57 us; skew 500 ppm
I20260927 18:39:19.217003 11239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:19 dist-test-slave-9jll krb5kdc[10720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534359, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:19.220161 11239 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:39:19.221764 11239 webserver.cc:460] Webserver started at http://127.2.95.126:36003/ using document root <none> and password file <none>
I20260927 18:39:19.222481 11239 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:19.222608 11239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:19.228638 11239 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 18:39:19.231789 11254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:19.233352 11239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:39:19.233533 11239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "8c143fbbb1a4464899efb3c015f717db"
format_stamp: "Formatted at 2026-09-27 18:39:16 on dist-test-slave-9jll"
server_key: "1e8ed3ab5f3243bac3f0f76fead6b4d0"
server_key_iv: "92dee1e1e24b02e42183f755d02fb311"
server_key_version: "encryptionkey@0"
I20260927 18:39:19.234553 11239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:19.262167 11239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:19.266626 11239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:19.266876 11239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:19.284113 11239 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:38773
I20260927 18:39:19.284150 11305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:38773 every 8 connection(s)
I20260927 18:39:19.285537 11239 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:39:19.290729 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 11239
I20260927 18:39:19.291501 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:32915
--local_ip_for_outbound_sockets=127.2.95.65
--tserver_master_addrs=127.2.95.126:38773
--webserver_port=42235
--webserver_interface=127.2.95.65
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.95.65
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:39:19.292516 11306 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:39:19.295817 11306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db: Bootstrap starting.
I20260927 18:39:19.314502 11306 log.cc:824] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:19.323964 11306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:39:19.324534 11306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db: Bootstrap complete.
I20260927 18:39:19.333724 11306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c143fbbb1a4464899efb3c015f717db" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38773 } }
I20260927 18:39:19.334635 11306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c143fbbb1a4464899efb3c015f717db, State: Initialized, Role: FOLLOWER
I20260927 18:39:19.335223 11306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c143fbbb1a4464899efb3c015f717db" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38773 } }
I20260927 18:39:19.335453 11306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:39:19.335557 11306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:39:19.335726 11306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:39:19.339641 11306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c143fbbb1a4464899efb3c015f717db" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38773 } }
I20260927 18:39:19.340174 11306 leader_election.cc:302] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c143fbbb1a4464899efb3c015f717db; no voters:
I20260927 18:39:19.340764 11306 leader_election.cc:288] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:39:19.340899 11310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:39:19.343106 11306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:39:19.348048 11310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [term 2 LEADER]: Becoming Leader. State: Replica: 8c143fbbb1a4464899efb3c015f717db, State: Running, Role: LEADER
I20260927 18:39:19.348757 11310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c143fbbb1a4464899efb3c015f717db" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38773 } }
I20260927 18:39:19.351526 11311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8c143fbbb1a4464899efb3c015f717db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c143fbbb1a4464899efb3c015f717db" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38773 } } }
I20260927 18:39:19.351984 11311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:19.352540 11312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c143fbbb1a4464899efb3c015f717db. Latest consensus state: current_term: 2 leader_uuid: "8c143fbbb1a4464899efb3c015f717db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c143fbbb1a4464899efb3c015f717db" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38773 } } }
I20260927 18:39:19.352828 11312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:19.354961 11322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:39:19.367936 11322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:39:19.370092 11322 catalog_manager.cc:1259] Loaded cluster ID: 6346daac90e4464da609e94c10367004
I20260927 18:39:19.370324 11322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:39:19.375191 11322 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:39:19.377174 11322 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8c143fbbb1a4464899efb3c015f717db: Loaded TSK: 0
I20260927 18:39:19.379168 11322 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:39:19.663900 11308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:19.664206 11308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:19.664340 11308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:19.671941 11308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:19.672072 11308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:19.672219 11308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:39:19.682338 11308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.65.keytab
--principal=oryx/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:32915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=42235
--tserver_master_addrs=127.2.95.126:38773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:19.683696 11308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:19.685002 11308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:19.698764 11333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:19.698735 11334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:19.699656 11336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:19.700014 11308 server_base.cc:1034] running on GCE node
I20260927 18:39:19.700649 11308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:19.701767 11308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:19.702991 11308 hybrid_clock.cc:648] HybridClock initialized: now 1790534359702931 us; error 50 us; skew 500 ppm
I20260927 18:39:19.703296 11308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:19 dist-test-slave-9jll krb5kdc[10720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534359, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:19.706487 11308 init.cc:377] Logged in from keytab as oryx/127.2.95.65@KRBTEST.COM (short username oryx)
I20260927 18:39:19.708057 11308 webserver.cc:460] Webserver started at http://127.2.95.65:42235/ using document root <none> and password file <none>
I20260927 18:39:19.708735 11308 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:19.708854 11308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:19.714383 11308 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:39:19.717152 11343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:19.718626 11308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:39:19.718780 11308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "2bc7181be9ed4cfda98375c7d75faa72"
format_stamp: "Formatted at 2026-09-27 18:39:17 on dist-test-slave-9jll"
server_key: "34b7be42cea4402a39086be54b9ce52f"
server_key_iv: "b6b9a1c608be1d0f88e8aea95658a6ac"
server_key_version: "encryptionkey@0"
I20260927 18:39:19.719719 11308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:19.761785 11308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:19.766168 11308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:19.766462 11308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:19.767477 11308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:19.768860 11308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:19.768960 11308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:19.769061 11308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:19.769134 11308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:19.800907 11308 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:32915
I20260927 18:39:19.800984 11455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:32915 every 8 connection(s)
I20260927 18:39:19.802439 11308 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:39:19.803930 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 11308
I20260927 18:39:19.804694 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:43289
--local_ip_for_outbound_sockets=127.2.95.66
--tserver_master_addrs=127.2.95.126:38773
--webserver_port=42455
--webserver_interface=127.2.95.66
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.95.66
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:39:19 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:19 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:19.830267 11459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:19.806219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:44629)
0927 18:39:19.807884 (+ 1665us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:19.807945 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:19.810211 (+ 2266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:19.811807 (+ 1596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:19.811815 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:19.812327 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:19.812919 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:19.812946 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:19.815169 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:19.815173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:19.816557 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:19.816572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:19.817968 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:19.817973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:19.818182 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:19.824364 (+ 6182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:19.824416 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:19.828851 (+ 4435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1376,"thread_start_us":1167,"threads_started":1}
W20260927 18:39:19.830549 11458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:19.806161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:44629 (local address 127.2.95.126:38773)
0927 18:39:19.807271 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:19.807287 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:19.808109 (+ 822us) server_negotiation.cc:407] Connection header received
0927 18:39:19.810852 (+ 2743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:19.810866 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:19.811266 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:19.811682 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:19.813137 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:19.814987 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:19.816709 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:19.817663 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:19.817894 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:19.829236 (+ 11342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:44629: BlockingRecv error: recv got EOF from 127.2.95.65:44629 (error 108)
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":330,"threads_started":1}
W20260927 18:39:19.831468 11456 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:38773 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.95.126:38773: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Sep 27 18:39:19 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:19 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:19.840113 11459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:19.832055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:47037)
0927 18:39:19.832246 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:19.832268 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:19.832370 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:19.832789 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:19.832794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:19.833140 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:19.833282 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:19.833291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:19.834372 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:19.834377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:19.835275 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:19.835287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:19.835966 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:19.835973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:19.836149 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:19.836692 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:19.836710 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:19.839828 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 18:39:19.840415 11458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:19.832238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:47037 (local address 127.2.95.126:38773)
0927 18:39:19.832534 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:19.832538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:19.832554 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:39:19.832601 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:19.832604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:19.832660 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:19.832742 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:19.833436 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:19.834223 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:19.835415 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:19.835820 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:19.835932 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:19.840075 (+ 4143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:47037: BlockingRecv error: recv got EOF from 127.2.95.65:47037 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 27 18:39:19 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:19 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:19.848541 11459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:19.840821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:55195)
0927 18:39:19.841001 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:19.841021 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:19.841134 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:19.841481 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:19.841485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:19.841857 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:19.842015 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:19.842025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:19.843009 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:19.843014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:19.843999 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:19.844011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:19.844730 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:19.844736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:19.844922 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:19.845499 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:19.845522 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:19.848218 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 18:39:19.848699 11458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:19.840899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:55195 (local address 127.2.95.126:38773)
0927 18:39:19.841137 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:19.841142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:19.841161 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:39:19.841273 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:19.841278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:19.841356 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:19.841435 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:19.842154 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:19.842870 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:19.844153 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:19.844585 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:19.844724 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:19.848471 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:55195: BlockingRecv error: recv got EOF from 127.2.95.65:55195 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
W20260927 18:39:19.848939 11456 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:39:20.148478 11460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:20.148772 11460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:20.148904 11460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:20.156347 11460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:20.156473 11460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:20.156615 11460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:39:20.166538 11460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.66.keytab
--principal=oryx/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:43289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=42455
--tserver_master_addrs=127.2.95.126:38773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:20.167860 11460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:20.169170 11460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:20.183223 11466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:20.183149 11465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:20.183571 11468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:20.187435 11460 server_base.cc:1034] running on GCE node
I20260927 18:39:20.188199 11460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:20.189270 11460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:20.190503 11460 hybrid_clock.cc:648] HybridClock initialized: now 1790534360190454 us; error 82 us; skew 500 ppm
I20260927 18:39:20.190862 11460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534360, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:20.194402 11460 init.cc:377] Logged in from keytab as oryx/127.2.95.66@KRBTEST.COM (short username oryx)
I20260927 18:39:20.196487 11460 webserver.cc:460] Webserver started at http://127.2.95.66:42455/ using document root <none> and password file <none>
I20260927 18:39:20.197494 11460 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:20.197654 11460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:20.203667 11460 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 18:39:20.206780 11475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:20.208362 11460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:39:20.208576 11460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "a6d8cd8bb7204aedb789e7594433858d"
format_stamp: "Formatted at 2026-09-27 18:39:17 on dist-test-slave-9jll"
server_key: "c54239e6b0878c0ae60a874daf4dff6b"
server_key_iv: "d53d99795ab706b4ada977c6e30d774b"
server_key_version: "encryptionkey@0"
I20260927 18:39:20.209529 11460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:20.226909 11460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:20.231523 11460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:20.231792 11460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:20.232807 11460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:20.234241 11460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:20.234336 11460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:20.234448 11460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:20.234521 11460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:20.264917 11460 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:43289
I20260927 18:39:20.264966 11587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:43289 every 8 connection(s)
I20260927 18:39:20.266355 11460 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:39:20.270476 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 11460
I20260927 18:39:20.271145 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:41763
--local_ip_for_outbound_sockets=127.2.95.67
--tserver_master_addrs=127.2.95.126:38773
--webserver_port=46047
--webserver_interface=127.2.95.67
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.95.67
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:39:20.282876 11458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:20.268883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:53295 (local address 127.2.95.126:38773)
0927 18:39:20.269126 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:20.269131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:20.270227 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 18:39:20.272063 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:20.272068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:20.272150 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:20.272234 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:20.273710 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.274424 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.276038 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.276477 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.276594 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.282637 (+ 6043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:53295: BlockingRecv error: recv got EOF from 127.2.95.66:53295 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:39:20.283483 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:20.269286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:53295)
0927 18:39:20.270068 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:20.270112 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:20.271789 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:20.272474 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:20.272483 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:20.272970 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:20.273512 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.273540 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.274587 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.274591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.275880 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.275894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.276726 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.276734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.277090 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.277953 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:20.277984 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:20.282383 (+ 4399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":319,"threads_started":1}
W20260927 18:39:20.284327 11588 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:38773 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.95.126:38773: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:20.292753 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:20.284934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:59005)
0927 18:39:20.285198 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:20.285218 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:20.285348 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:20.285653 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:20.285658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:20.286041 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:20.286183 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.286192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.287211 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.287217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.288206 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.288220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.288951 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.288957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.289131 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.289714 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:20.289739 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:20.292481 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142}
W20260927 18:39:20.292984 11458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:20.285050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:59005 (local address 127.2.95.126:38773)
0927 18:39:20.285230 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:20.285236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:20.285278 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:39:20.285465 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:20.285470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:20.285531 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:20.285611 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:20.286324 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.287074 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.288346 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.288803 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.288912 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.292734 (+ 3822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:59005: BlockingRecv error: recv got EOF from 127.2.95.66:59005 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:20.302006 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:20.293758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:45159)
0927 18:39:20.294085 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:20.294104 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:20.294208 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:20.294568 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:20.294572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:20.294877 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:20.295036 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.295045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.296070 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.296074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.296996 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.297008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.297757 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.297763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.297973 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.298518 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:20.298536 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:20.301701 (+ 3165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195}
W20260927 18:39:20.302325 11588 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:39:20.302175 11458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:20.294163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:45159 (local address 127.2.95.126:38773)
0927 18:39:20.294318 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:20.294324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:20.294337 (+ 13us) server_negotiation.cc:407] Connection header received
0927 18:39:20.294382 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:20.294386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:20.294438 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:20.294524 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:20.295174 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.295933 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.297136 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.297605 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.297743 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.301978 (+ 4235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:45159: BlockingRecv error: recv got EOF from 127.2.95.66:45159 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 18:39:20.603941 11591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:20.604246 11591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:20.604377 11591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:20.611909 11591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:20.612056 11591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:20.612227 11591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:39:20.622406 11591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/krb5kdc/oryx_127.2.95.67.keytab
--principal=oryx/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:41763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=46047
--tserver_master_addrs=127.2.95.126:38773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:20.623827 11591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:20.625258 11591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:20.638868 11599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:20.639061 11597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:20.638912 11596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:20.639560 11591 server_base.cc:1034] running on GCE node
I20260927 18:39:20.640192 11591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:20.641177 11591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:20.642429 11591 hybrid_clock.cc:648] HybridClock initialized: now 1790534360642358 us; error 62 us; skew 500 ppm
I20260927 18:39:20.642774 11591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534360, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:20.646143 11591 init.cc:377] Logged in from keytab as oryx/127.2.95.67@KRBTEST.COM (short username oryx)
I20260927 18:39:20.647693 11591 webserver.cc:460] Webserver started at http://127.2.95.67:46047/ using document root <none> and password file <none>
I20260927 18:39:20.648466 11591 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:20.648599 11591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:20.654500 11591 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:39:20.657295 11606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:20.658728 11591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 18:39:20.658897 11591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "50076f866b064a79a776e859c6642ccd"
format_stamp: "Formatted at 2026-09-27 18:39:18 on dist-test-slave-9jll"
server_key: "3fea02b5790e709ef940aa25568de467"
server_key_iv: "3b7e1b2da08ca5e878fd911e26293430"
server_key_version: "encryptionkey@0"
I20260927 18:39:20.659876 11591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:20.677650 11591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:20.682354 11591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:20.682713 11591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:20.683707 11591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:20.685075 11591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:20.685163 11591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:20.685256 11591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:20.685328 11591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:20.718227 11591 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:41763
I20260927 18:39:20.718308 11718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:41763 every 8 connection(s)
I20260927 18:39:20.719698 11591 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:39:20.726482 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 11591
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 18:39:20.738487 11458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:20.722425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:45095 (local address 127.2.95.126:38773)
0927 18:39:20.722658 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:20.722663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:20.723967 (+ 1304us) server_negotiation.cc:407] Connection header received
0927 18:39:20.725895 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:20.725900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:20.725984 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:20.726100 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:20.728271 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.729325 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.731773 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.732418 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.732555 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.738180 (+ 5625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:45095: BlockingRecv error: recv got EOF from 127.2.95.67:45095 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:39:20.739375 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:20.722940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:45095)
0927 18:39:20.723766 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:20.723822 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:20.725546 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:20.726383 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:20.726393 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:20.726983 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:20.727908 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.727937 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.729496 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.729500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.731623 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.731639 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.732666 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.732672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.732904 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.733699 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:20.733736 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:20.737968 (+ 4232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":332,"threads_started":1}
W20260927 18:39:20.740175 11719 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:38773 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.95.126:38773: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
I20260927 18:39:20.744962 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:20.727511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43808 (local address 127.2.95.126:38773)
0927 18:39:20.728289 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:20.728293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:20.728305 (+ 12us) server_negotiation.cc:407] Connection header received
0927 18:39:20.728353 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:20.728356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:20.728413 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:20.728494 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:20.730340 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.731434 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.733128 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.733700 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.733833 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.736668 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:20.736682 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:20.736706 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:20.736757 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:20.740948 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:20.741945 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:20.741953 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:20.741960 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:20.742342 (+ 382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:20.742755 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:20.742760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:20.742764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:20.743437 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:20.743691 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:20.744252 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 18:39:20.744581 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":559,"threads_started":1}
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:20.749727 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:20.740739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:57251)
0927 18:39:20.740902 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:20.740920 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:20.741014 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:20.741296 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:20.741301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:20.741740 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:20.741940 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.741954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.743284 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.743288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.744641 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.744656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.745560 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.745565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.745753 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.746335 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:20.746355 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:20.749421 (+ 3066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 18:39:20.750021 11458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:20.740854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:57251 (local address 127.2.95.126:38773)
0927 18:39:20.741021 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:20.741038 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:20.741056 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:39:20.741108 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:20.741113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:20.741175 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:20.741244 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:20.742089 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.743147 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.744780 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.745409 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.745532 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.749642 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:57251: BlockingRecv error: recv got EOF from 127.2.95.67:57251 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:20.757987 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:20.750383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:43669)
0927 18:39:20.750597 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:20.750617 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:20.750738 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:20.751093 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:20.751097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:20.751392 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:20.751542 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.751550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.752492 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.752497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.753480 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.753494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.754291 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.754295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.754450 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.754993 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:20.755010 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:20.757657 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 18:39:20.758141 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:20.750474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:43669 (local address 127.2.95.126:38773)
0927 18:39:20.750728 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:20.750732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:20.750748 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:39:20.750822 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:20.750826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:20.750892 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:20.750966 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:20.751672 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.752361 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.753640 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.754133 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.754232 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.757922 (+ 3690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:43669: BlockingRecv error: recv got EOF from 127.2.95.67:43669 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
W20260927 18:39:20.758347 11719 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:20 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:20.858259 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:20.849466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:53027)
0927 18:39:20.850093 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:20.850110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:20.850238 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:20.850545 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:20.850548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:20.850976 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:20.851218 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.851229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.852316 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.852319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.853307 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:20.853319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.854099 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.854104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:20.854248 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.854867 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:20.854885 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:20.857926 (+ 3041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":360,"threads_started":1}
W20260927 18:39:20.858397 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:20.849584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:53027 (local address 127.2.95.126:38773)
0927 18:39:20.849784 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:20.849788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:20.850211 (+ 423us) server_negotiation.cc:407] Connection header received
0927 18:39:20.850356 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:20.850360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:20.850426 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:20.850517 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:20.851380 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.852179 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.853430 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:20.853885 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:20.853983 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:20.858162 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:53027: BlockingRecv error: recv got EOF from 127.2.95.65:53027 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 18:39:21 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:21 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:21.311777 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:21.302931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:46613)
0927 18:39:21.303411 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:21.303428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:21.303561 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:21.303893 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:21.303897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:21.304463 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:21.304657 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:21.304666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:21.305789 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:21.305793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:21.306814 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:21.306828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:21.307583 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:21.307588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:21.307721 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:21.308412 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:21.308430 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:21.311542 (+ 3112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":265,"threads_started":1}
W20260927 18:39:21.311956 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:21.303039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:46613 (local address 127.2.95.126:38773)
0927 18:39:21.303244 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:21.303251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:21.303545 (+ 294us) server_negotiation.cc:407] Connection header received
0927 18:39:21.303662 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:21.303667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:21.303732 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:21.303817 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:21.304815 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:21.305574 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:21.306942 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:21.307386 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:21.307477 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:21.311750 (+ 4273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:46613: BlockingRecv error: recv got EOF from 127.2.95.66:46613 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 18:39:21 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:21 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:21.767731 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:21.759068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:60545)
0927 18:39:21.759591 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:21.759612 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:21.759736 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:21.760261 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:21.760265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:21.760716 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:21.760954 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:21.760968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:21.762173 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:21.762176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:21.763146 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:21.763158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:21.763975 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:21.763982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:21.764114 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:21.764723 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:21.764745 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:21.767499 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":278,"threads_started":1}
W20260927 18:39:21.768014 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:21.759268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:60545 (local address 127.2.95.126:38773)
0927 18:39:21.759458 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:21.759464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:21.759708 (+ 244us) server_negotiation.cc:407] Connection header received
0927 18:39:21.759855 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:21.759859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:21.759947 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:21.760034 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:21.761105 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:21.762052 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:21.763285 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:21.763718 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:21.763813 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:21.767755 (+ 3942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:60545: BlockingRecv error: recv got EOF from 127.2.95.67:60545 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 18:39:21 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:21 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:21.868201 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:21.859431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:48363)
0927 18:39:21.860058 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:21.860085 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:21.860262 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:21.860685 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:21.860691 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:21.861085 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:21.861231 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:21.861240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:21.862353 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:21.862357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:21.863299 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:21.863313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:21.864035 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:21.864041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:21.864193 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:21.864822 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:21.864842 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:21.867961 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":330,"threads_started":1}
W20260927 18:39:21.868407 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:21.859599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:48363 (local address 127.2.95.126:38773)
0927 18:39:21.859797 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:21.859803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:21.860209 (+ 406us) server_negotiation.cc:407] Connection header received
0927 18:39:21.860414 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:21.860418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:21.860487 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:21.860576 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:21.861377 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:21.862215 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:21.863442 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:21.863885 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:21.863995 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:21.868191 (+ 4196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:48363: BlockingRecv error: recv got EOF from 127.2.95.65:48363 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 18:39:22 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:22 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:22.321863 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:22.312898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:35899)
0927 18:39:22.313591 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:22.313608 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:22.313781 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:22.314112 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:22.314118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:22.314535 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:22.314719 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:22.314733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:22.315730 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:22.315736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:22.316628 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:22.316640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:22.317353 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:22.317358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:22.317524 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:22.318134 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:22.318153 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:22.321438 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":449,"threads_started":1}
W20260927 18:39:22.322047 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:22.313113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:35899 (local address 127.2.95.126:38773)
0927 18:39:22.313321 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:22.313327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:22.313696 (+ 369us) server_negotiation.cc:407] Connection header received
0927 18:39:22.313900 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:22.313905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:22.313974 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:22.314053 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:22.314857 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:22.315594 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:22.316771 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:22.317205 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:22.317315 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:22.321752 (+ 4437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:35899: BlockingRecv error: recv got EOF from 127.2.95.66:35899 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 18:39:22 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:22 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:22.777099 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:22.768707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:60443)
0927 18:39:22.769284 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:22.769300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:22.769430 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:22.769798 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:22.769803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:22.770205 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:22.770358 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:22.770368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:22.771326 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:22.771331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:22.772222 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:22.772234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:22.772942 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:22.772948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:22.773092 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:22.773663 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:22.773706 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:22.776780 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":343,"threads_started":1}
W20260927 18:39:22.777256 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:22.768878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:60443 (local address 127.2.95.126:38773)
0927 18:39:22.769126 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:22.769132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:22.769384 (+ 252us) server_negotiation.cc:407] Connection header received
0927 18:39:22.769563 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:22.769568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:22.769641 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:22.769748 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:22.770504 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:22.771198 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:22.772359 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:22.772789 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:22.772896 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:22.777017 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:60443: BlockingRecv error: recv got EOF from 127.2.95.67:60443 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 18:39:22 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:22 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:22.877722 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:22.869253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:47491)
0927 18:39:22.869784 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:22.869800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:22.869928 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:22.870233 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:22.870236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:22.870551 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:22.870746 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:22.870755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:22.871823 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:22.871827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:22.872708 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:22.872719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:22.873389 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:22.873393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:22.873522 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:22.874153 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:22.874170 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:22.877375 (+ 3205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":250,"threads_started":1}
W20260927 18:39:22.877841 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:22.869344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:47491 (local address 127.2.95.126:38773)
0927 18:39:22.869533 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:22.869537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:22.869882 (+ 345us) server_negotiation.cc:407] Connection header received
0927 18:39:22.870024 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:22.870028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:22.870097 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:22.870192 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:22.870867 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:22.871680 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:22.872838 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:22.873252 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:22.873349 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:22.877583 (+ 4234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:47491: BlockingRecv error: recv got EOF from 127.2.95.65:47491 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 18:39:23 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:23 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:23.332026 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:23.323222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:49675)
0927 18:39:23.323683 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:23.323700 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:23.323820 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:23.324216 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:23.324220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:23.324612 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:23.324803 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:23.324816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:23.326149 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:23.326153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:23.327063 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:23.327076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:23.327784 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:23.327789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:23.328002 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:23.328633 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:23.328650 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:23.331754 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":246,"threads_started":1}
W20260927 18:39:23.332221 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:23.323225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:49675 (local address 127.2.95.126:38773)
0927 18:39:23.323380 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:23.323384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:23.323786 (+ 402us) server_negotiation.cc:407] Connection header received
0927 18:39:23.323923 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:23.323928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:23.324004 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:23.324171 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:23.324942 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:23.326024 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:23.327212 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:23.327634 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:23.327746 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:23.332020 (+ 4274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:49675: BlockingRecv error: recv got EOF from 127.2.95.66:49675 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 18:39:23 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:23 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:23.786707 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:23.778297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:51263)
0927 18:39:23.778784 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:23.778801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:23.778922 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:23.779344 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:23.779348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:23.779697 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:23.779876 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:23.779886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:23.780972 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:23.780976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:23.781967 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:23.781980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:23.782697 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:23.782701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:23.782830 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:23.783533 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:23.783557 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:23.786455 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":242,"threads_started":1}
W20260927 18:39:23.786937 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:23.778426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:51263 (local address 127.2.95.126:38773)
0927 18:39:23.778679 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:23.778685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:23.778908 (+ 223us) server_negotiation.cc:407] Connection header received
0927 18:39:23.779093 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:23.779097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:23.779181 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:23.779270 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:23.780087 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:23.780827 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:23.782100 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:23.782538 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:23.782661 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:23.786662 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:51263: BlockingRecv error: recv got EOF from 127.2.95.67:51263 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 27 18:39:23 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:23 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:23.887084 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:23.878722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:42333)
0927 18:39:23.879253 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:23.879273 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:23.879402 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:23.879718 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:23.879722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:23.880087 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:23.880233 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:23.880241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:23.881410 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:23.881414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:23.882392 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:23.882406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:23.883100 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:23.883105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:23.883260 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:23.883816 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:23.883834 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:23.886750 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":305,"threads_started":1}
W20260927 18:39:23.887200 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:23.878837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:42333 (local address 127.2.95.126:38773)
0927 18:39:23.879012 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:23.879017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:23.879359 (+ 342us) server_negotiation.cc:407] Connection header received
0927 18:39:23.879501 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:23.879505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:23.879570 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:23.879678 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:23.880376 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:23.881281 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:23.882533 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:23.882952 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:23.883064 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:23.886998 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:42333: BlockingRecv error: recv got EOF from 127.2.95.65:42333 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 18:39:24 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:24 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:24.341727 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:24.333022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:37121)
0927 18:39:24.333595 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:24.333620 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:24.333820 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:24.334197 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:24.334202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:24.334598 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:24.334752 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:24.334761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:24.335810 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:24.335816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:24.336790 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:24.336806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:24.337529 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:24.337535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:24.337752 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:24.338474 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:24.338492 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:24.341402 (+ 2910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":333,"threads_started":1}
W20260927 18:39:24.341984 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:24.333229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:37121 (local address 127.2.95.126:38773)
0927 18:39:24.333455 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:24.333461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:24.333729 (+ 268us) server_negotiation.cc:407] Connection header received
0927 18:39:24.333953 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:24.333958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:24.334037 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:24.334137 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:24.334901 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:24.335661 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:24.336942 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:24.337370 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:24.337489 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:24.341699 (+ 4210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:37121: BlockingRecv error: recv got EOF from 127.2.95.66:37121 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 18:39:24 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:24 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:24.795938 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:24.787653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:38829)
0927 18:39:24.788118 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:24.788135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:24.788283 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:24.788655 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:24.788658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:24.789038 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:24.789190 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:24.789199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:24.790279 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:24.790284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:24.791330 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:24.791343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:24.792121 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:24.792128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:24.792289 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:24.792829 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:24.792846 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:24.795705 (+ 2859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":239,"threads_started":1}
W20260927 18:39:24.796166 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:24.787765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:38829 (local address 127.2.95.126:38773)
0927 18:39:24.787947 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:24.787952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:24.788233 (+ 281us) server_negotiation.cc:407] Connection header received
0927 18:39:24.788399 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:24.788403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:24.788484 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:24.788566 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:24.789336 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:24.790152 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:24.791480 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:24.791883 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:24.791992 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:24.795940 (+ 3948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:38829: BlockingRecv error: recv got EOF from 127.2.95.67:38829 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 18:39:24 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:24 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:24.905359 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:24.888146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:53191)
0927 18:39:24.888714 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:24.888729 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:24.888865 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:24.889299 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:24.889303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:24.889771 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:24.889960 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:24.889971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:24.891112 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:24.891117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:24.892404 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:24.892417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:24.893103 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:24.893108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:24.893232 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:24.901422 (+ 8190us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:24.901447 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:24.905117 (+ 3670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":315,"threads_started":1}
W20260927 18:39:24.905524 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:24.888327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:53191 (local address 127.2.95.126:38773)
0927 18:39:24.888622 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:24.888628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:24.888810 (+ 182us) server_negotiation.cc:407] Connection header received
0927 18:39:24.889090 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:24.889095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:24.889158 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:24.889246 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:24.890128 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:24.890967 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:24.892552 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:24.892966 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:24.893064 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:24.905315 (+ 12251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:53191: BlockingRecv error: recv got EOF from 127.2.95.65:53191 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 27 18:39:25 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:25 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:25.351881 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:25.343426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:38233)
0927 18:39:25.343987 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:25.344006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:25.344144 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:25.344485 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:25.344490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:25.344945 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:25.345107 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:25.345117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:25.346202 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:25.346206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:25.347102 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:25.347114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:25.347880 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:25.347884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:25.348018 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:25.348598 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:25.348615 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:25.351560 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":255,"threads_started":1}
W20260927 18:39:25.352025 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:25.343632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:38233 (local address 127.2.95.126:38773)
0927 18:39:25.343797 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:25.343802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:25.344079 (+ 277us) server_negotiation.cc:407] Connection header received
0927 18:39:25.344256 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:25.344261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:25.344335 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:25.344419 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:25.345257 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:25.346094 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:25.347248 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:25.347737 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:25.347840 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:25.351830 (+ 3990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:38233: BlockingRecv error: recv got EOF from 127.2.95.66:38233 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 18:39:25 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:25 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:25.806457 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:25.796933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:59511)
0927 18:39:25.797474 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:25.797491 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:25.797614 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:25.797946 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:25.797950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:25.798411 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:25.798571 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:25.798581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:25.799727 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:25.799732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:25.800783 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:25.800798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:25.801705 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:25.801711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:25.801890 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:25.802560 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:25.802583 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:25.806138 (+ 3555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":291,"threads_started":1}
W20260927 18:39:25.806608 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:25.797085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:59511 (local address 127.2.95.126:38773)
0927 18:39:25.797340 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:25.797346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:25.797564 (+ 218us) server_negotiation.cc:407] Connection header received
0927 18:39:25.797746 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:25.797750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:25.797828 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:25.797912 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:25.798730 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:25.799557 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:25.800949 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:25.801510 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:25.801625 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:25.806366 (+ 4741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:59511: BlockingRecv error: recv got EOF from 127.2.95.67:59511 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 27 18:39:25 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:25 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:25.914222 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:25.906297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:52841)
0927 18:39:25.906792 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:25.906808 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:25.906919 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:25.907204 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:25.907208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:25.907565 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:25.907719 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:25.907727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:25.908803 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:25.908808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:25.909799 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:25.909813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:25.910498 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:25.910502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:25.910621 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:25.911173 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:25.911190 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:25.913973 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":292,"threads_started":1}
W20260927 18:39:25.914482 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:25.906374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:52841 (local address 127.2.95.126:38773)
0927 18:39:25.906525 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:25.906530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:25.906875 (+ 345us) server_negotiation.cc:407] Connection header received
0927 18:39:25.907015 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:25.907019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:25.907077 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:25.907157 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:25.907847 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:25.908649 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:25.909946 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:25.910367 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:25.910461 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:25.914250 (+ 3789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:52841: BlockingRecv error: recv got EOF from 127.2.95.65:52841 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 18:39:26 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:26 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:26.362273 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:26.352952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:54913)
0927 18:39:26.353493 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:26.353512 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:26.353635 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:26.354073 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:26.354077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:26.354459 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:26.354638 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:26.354648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:26.355742 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:26.355747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:26.357003 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:26.357020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:26.358123 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:26.358129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:26.358264 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:26.358913 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:26.358932 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:26.361970 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":335,"threads_started":1}
W20260927 18:39:26.362452 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:26.353059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:54913 (local address 127.2.95.126:38773)
0927 18:39:26.353275 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:26.353280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:26.353589 (+ 309us) server_negotiation.cc:407] Connection header received
0927 18:39:26.353762 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:26.353766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:26.353833 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:26.353925 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:26.354800 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:26.355557 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:26.357241 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:26.357864 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:26.357974 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:26.362221 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:54913: BlockingRecv error: recv got EOF from 127.2.95.66:54913 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 18:39:26 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:26 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:26.815681 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:26.807580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:46867)
0927 18:39:26.808145 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:26.808162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:26.808294 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:26.808639 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:26.808643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:26.809012 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:26.809159 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:26.809168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:26.810297 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:26.810303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:26.811198 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:26.811209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:26.811895 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:26.811900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:26.812060 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:26.812625 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:26.812644 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:26.815373 (+ 2729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":283,"threads_started":1}
W20260927 18:39:26.815815 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:26.807737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:46867 (local address 127.2.95.126:38773)
0927 18:39:26.808048 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:26.808055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:26.808244 (+ 189us) server_negotiation.cc:407] Connection header received
0927 18:39:26.808411 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:26.808416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:26.808500 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:26.808598 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:26.809297 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:26.810167 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:26.811322 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:26.811751 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:26.811856 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:26.815614 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:46867: BlockingRecv error: recv got EOF from 127.2.95.67:46867 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 18:39:26 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:26 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:26.924376 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:26.915241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:41111)
0927 18:39:26.915805 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:26.915821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:26.915946 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:26.916342 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:26.916345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:26.916777 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:26.916943 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:26.916953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:26.918134 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:26.918139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:26.919118 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:26.919133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:26.919800 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:26.919804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:26.919918 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:26.920519 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:26.920538 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:26.924069 (+ 3531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":249,"threads_started":1}
W20260927 18:39:26.924645 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:26.915293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:41111 (local address 127.2.95.126:38773)
0927 18:39:26.915493 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:26.915500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:26.915906 (+ 406us) server_negotiation.cc:407] Connection header received
0927 18:39:26.916101 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:26.916105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:26.916175 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:26.916267 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:26.917106 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:26.917975 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:26.919255 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:26.919673 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:26.919772 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:26.924392 (+ 4620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:41111: BlockingRecv error: recv got EOF from 127.2.95.65:41111 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 18:39:27 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:27 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:27.372012 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:27.363370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:54325)
0927 18:39:27.363920 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:27.363937 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:27.364053 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:27.364475 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:27.364478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:27.364866 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:27.365029 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:27.365039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:27.366082 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:27.366087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:27.367054 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:27.367068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:27.367796 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:27.367801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:27.367924 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:27.368527 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:27.368545 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:27.371621 (+ 3076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":243,"threads_started":1}
W20260927 18:39:27.372061 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:27.363531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:54325 (local address 127.2.95.126:38773)
0927 18:39:27.363735 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:27.363740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:27.364019 (+ 279us) server_negotiation.cc:407] Connection header received
0927 18:39:27.364190 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:27.364194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:27.364260 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:27.364364 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:27.365168 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:27.365931 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:27.367218 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:27.367661 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:27.367758 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:27.371857 (+ 4099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:54325: BlockingRecv error: recv got EOF from 127.2.95.66:54325 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 18:39:27 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:27 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:27.826066 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:27.816734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:50871)
0927 18:39:27.817385 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:27.817404 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:27.817561 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:27.817985 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:27.817989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:27.818411 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:27.818576 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:27.818586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:27.819755 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:27.819760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:27.820782 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:27.820796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:27.821721 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:27.821727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:27.821866 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:27.822598 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:27.822622 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:27.825781 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":358,"threads_started":1}
W20260927 18:39:27.826246 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:27.816844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:50871 (local address 127.2.95.126:38773)
0927 18:39:27.817074 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:27.817081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:27.817536 (+ 455us) server_negotiation.cc:407] Connection header received
0927 18:39:27.817723 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:27.817727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:27.817809 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:27.817909 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:27.818746 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:27.819595 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:27.820944 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:27.821525 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:27.821653 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:27.825973 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:50871: BlockingRecv error: recv got EOF from 127.2.95.67:50871 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 18:39:27 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:27 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:27.934758 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:27.925340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:50639)
0927 18:39:27.925886 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:27.925902 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:27.926106 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:27.926455 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:27.926458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:27.926819 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:27.926965 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:27.926974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:27.928225 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:27.928230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:27.929154 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:27.929168 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:27.930002 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:27.930009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:27.930157 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:27.930811 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:27.930837 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:27.934488 (+ 3651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":257,"threads_started":1}
W20260927 18:39:27.935007 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:27.925544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:50639 (local address 127.2.95.126:38773)
0927 18:39:27.925748 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:27.925752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:27.925977 (+ 225us) server_negotiation.cc:407] Connection header received
0927 18:39:27.926248 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:27.926253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:27.926318 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:27.926407 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:27.927196 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:27.928042 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:27.929297 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:27.929786 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:27.929907 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:27.934743 (+ 4836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:50639: BlockingRecv error: recv got EOF from 127.2.95.65:50639 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 18:39:28 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:28 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:28.381742 11746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:28.373199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:42829)
0927 18:39:28.373824 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:28.373846 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:28.374008 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:28.374457 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:28.374462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:28.375019 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:28.375174 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:28.375184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:28.376393 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:28.376398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:28.377291 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:28.377303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:28.378142 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:28.378149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:28.378274 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:28.378809 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:28.378826 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:28.381463 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":336,"threads_started":1}
W20260927 18:39:28.381989 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:28.373482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:42829 (local address 127.2.95.126:38773)
0927 18:39:28.373729 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:28.373735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:28.373940 (+ 205us) server_negotiation.cc:407] Connection header received
0927 18:39:28.374133 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:28.374138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:28.374276 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:28.374368 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:28.375316 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:28.376224 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:28.377423 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:28.377975 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:28.378082 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:28.381741 (+ 3659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:42829: BlockingRecv error: recv got EOF from 127.2.95.66:42829 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 27 18:39:28 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:28 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:28.834980 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:28.826920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:47963)
0927 18:39:28.827418 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:28.827435 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:28.827548 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:28.827828 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:28.827830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:28.828160 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:28.828315 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:28.828324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:28.829291 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:28.829294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:28.830279 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:28.830293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:28.830980 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:28.830985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:28.831097 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:28.831637 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:28.831654 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:28.834688 (+ 3034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":272,"threads_started":1}
W20260927 18:39:28.835209 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:28.827024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:47963 (local address 127.2.95.126:38773)
0927 18:39:28.827204 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:28.827209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:28.827500 (+ 291us) server_negotiation.cc:407] Connection header received
0927 18:39:28.827647 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:28.827651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:28.827712 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:28.827794 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:28.828440 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:28.829183 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:28.830411 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:28.830820 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:28.830912 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:28.834904 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:47963: BlockingRecv error: recv got EOF from 127.2.95.67:47963 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 18:39:28 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:28 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:28.945430 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:28.935934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:39779)
0927 18:39:28.936551 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:28.936572 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:28.936708 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:28.937069 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:28.937075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:28.937594 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:28.937846 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:28.937857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:28.939203 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:28.939208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:28.940235 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:28.940249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:28.941021 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:28.941026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:28.941233 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:28.941895 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:28.941914 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:28.945032 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":336,"threads_started":1}
W20260927 18:39:28.945561 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:28.936467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:39779 (local address 127.2.95.126:38773)
0927 18:39:28.936741 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:28.936747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:28.936767 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:39:28.936821 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:28.936826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:28.936907 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:28.937007 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:28.938109 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:28.938924 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:28.940381 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:28.940868 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:28.940971 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:28.945290 (+ 4319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:39779: BlockingRecv error: recv got EOF from 127.2.95.65:39779 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 27 18:39:29 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:29 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:29.391502 11749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:29.382766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:36159)
0927 18:39:29.383439 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:29.383456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:29.383573 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:29.383910 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:29.383914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:29.384284 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:29.384453 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:29.384462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:29.385495 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:29.385499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:29.386467 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:29.386480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:29.387234 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:29.387238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:29.387367 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:29.387923 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:29.387940 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:29.391196 (+ 3256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":435,"threads_started":1}
W20260927 18:39:29.391584 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:29.382830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:36159 (local address 127.2.95.126:38773)
0927 18:39:29.383040 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:29.383047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:29.383535 (+ 488us) server_negotiation.cc:407] Connection header received
0927 18:39:29.383726 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:29.383730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:29.383794 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:29.383880 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:29.384617 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:29.385363 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:29.386609 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:29.387094 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:29.387196 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:29.391408 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:36159: BlockingRecv error: recv got EOF from 127.2.95.66:36159 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 18:39:29 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:29 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:29.845404 11750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:29.836182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:55017)
0927 18:39:29.836746 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:29.836762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:29.836914 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:29.837243 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:29.837248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:29.837721 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:29.837919 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:29.837932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:29.839278 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:29.839284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:29.840483 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:29.840499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:29.841289 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:29.841294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:29.841433 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:29.842048 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:29.842067 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:29.845060 (+ 2993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":302,"threads_started":1}
W20260927 18:39:29.845525 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:29.836425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:55017 (local address 127.2.95.126:38773)
0927 18:39:29.836657 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:29.836663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:29.836841 (+ 178us) server_negotiation.cc:407] Connection header received
0927 18:39:29.837031 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:29.837037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:29.837104 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:29.837187 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:29.838183 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:29.839088 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:29.840652 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:29.841133 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:29.841249 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:29.845297 (+ 4048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:55017: BlockingRecv error: recv got EOF from 127.2.95.67:55017 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 18:39:29 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:29 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:29.966652 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:29.946590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:43733)
0927 18:39:29.947092 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:29.947107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:29.947249 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:29.947564 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:29.947568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:29.947931 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:29.948079 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:29.948089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:29.949188 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:29.949194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:29.950262 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:29.950275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:29.950974 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:29.950980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:29.951128 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:29.962631 (+ 11503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:29.962656 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:29.966320 (+ 3664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":297,"threads_started":1}
W20260927 18:39:29.966986 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:29.946612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:43733 (local address 127.2.95.126:38773)
0927 18:39:29.946878 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:29.946885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:29.947195 (+ 310us) server_negotiation.cc:407] Connection header received
0927 18:39:29.947363 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:29.947369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:29.947439 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:29.947521 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:29.948218 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:29.949003 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:29.950399 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:29.950823 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:29.950945 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:29.966616 (+ 15671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:43733: BlockingRecv error: recv got EOF from 127.2.95.65:43733 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 18:39:30 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:30 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:30.401186 11752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:30.392435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:43783)
0927 18:39:30.392936 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:30.392952 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:30.393081 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:30.393395 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:30.393400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:30.393832 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:30.393975 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:30.393984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:30.395048 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:30.395052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:30.395976 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:30.395989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:30.396764 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:30.396770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:30.396940 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:30.397592 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:30.397609 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:30.400860 (+ 3251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":279,"threads_started":1}
W20260927 18:39:30.401314 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:30.392566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:43783 (local address 127.2.95.126:38773)
0927 18:39:30.392846 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:30.392852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:30.393032 (+ 180us) server_negotiation.cc:407] Connection header received
0927 18:39:30.393194 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:30.393200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:30.393266 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:30.393387 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:30.394122 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:30.394886 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:30.396146 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:30.396608 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:30.396729 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:30.401060 (+ 4331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:43783: BlockingRecv error: recv got EOF from 127.2.95.66:43783 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 27 18:39:30 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:30 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:30.854682 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:30.846492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:44027)
0927 18:39:30.846980 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:30.846996 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:30.847116 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:30.847444 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:30.847448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:30.847819 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:30.847959 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:30.847967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:30.849021 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:30.849026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:30.850074 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:30.850086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:30.850851 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:30.850856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:30.850968 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:30.851552 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:30.851570 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:30.854435 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":270,"threads_started":1}
W20260927 18:39:30.854848 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:30.846621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:44027 (local address 127.2.95.126:38773)
0927 18:39:30.846898 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:30.846905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:30.847073 (+ 168us) server_negotiation.cc:407] Connection header received
0927 18:39:30.847219 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:30.847223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:30.847300 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:30.847394 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:30.848082 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:30.848906 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:30.850201 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:30.850713 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:30.850833 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:30.854653 (+ 3820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:44027: BlockingRecv error: recv got EOF from 127.2.95.67:44027 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Sep 27 18:39:30 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:30 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:30.976374 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:30.967782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:53099)
0927 18:39:30.968328 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:30.968343 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:30.968482 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:30.968797 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:30.968802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:30.969210 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:30.969394 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:30.969403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:30.970714 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:30.970717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:30.971708 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:30.971722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:30.972410 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:30.972414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:30.972532 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:30.973237 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:30.973256 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:30.976114 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":308,"threads_started":1}
W20260927 18:39:30.976504 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:30.967898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:53099 (local address 127.2.95.126:38773)
0927 18:39:30.968121 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:30.968127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:30.968426 (+ 299us) server_negotiation.cc:407] Connection header received
0927 18:39:30.968606 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:30.968610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:30.968671 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:30.968756 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:30.969542 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:30.970575 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:30.971858 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:30.972282 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:30.972385 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:30.976291 (+ 3906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:53099: BlockingRecv error: recv got EOF from 127.2.95.65:53099 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 18:39:31 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:31 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:31.410861 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:31.402272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:47169)
0927 18:39:31.402798 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:31.402814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:31.402953 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:31.403335 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:31.403340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:31.403791 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:31.403968 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:31.403978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:31.405122 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:31.405127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:31.406163 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:31.406178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:31.406940 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:31.406946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:31.407107 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:31.407719 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:31.407737 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:31.410530 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":289,"threads_started":1}
W20260927 18:39:31.411020 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:31.402446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:47169 (local address 127.2.95.126:38773)
0927 18:39:31.402716 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:31.402722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:31.402899 (+ 177us) server_negotiation.cc:407] Connection header received
0927 18:39:31.403083 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:31.403088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:31.403157 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:31.403284 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:31.404120 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:31.404992 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:31.406305 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:31.406795 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:31.406904 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:31.410773 (+ 3869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:47169: BlockingRecv error: recv got EOF from 127.2.95.66:47169 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 18:39:31 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:31 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:31.864074 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:31.855685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:47141)
0927 18:39:31.856164 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:31.856183 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:31.856306 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:31.856592 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:31.856595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:31.857025 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:31.857201 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:31.857211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:31.858298 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:31.858302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:31.859237 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:31.859250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:31.860061 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:31.860066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:31.860211 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:31.860846 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:31.860863 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:31.863806 (+ 2943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":258,"threads_started":1}
W20260927 18:39:31.864403 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:31.855805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:47141 (local address 127.2.95.126:38773)
0927 18:39:31.855966 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:31.855970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:31.856269 (+ 299us) server_negotiation.cc:407] Connection header received
0927 18:39:31.856402 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:31.856406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:31.856471 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:31.856559 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:31.857372 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:31.858152 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:31.859405 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:31.859887 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:31.860018 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:31.864090 (+ 4072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:47141: BlockingRecv error: recv got EOF from 127.2.95.67:47141 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 18:39:31 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:31 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:31.985740 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:31.977536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:41793)
0927 18:39:31.978094 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:31.978110 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:31.978238 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:31.978632 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:31.978635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:31.979008 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:31.979172 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:31.979181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:31.980298 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:31.980302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:31.981244 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:31.981255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:31.982017 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:31.982021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:31.982153 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:31.982761 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:31.982779 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:31.985459 (+ 2680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":302,"threads_started":1}
W20260927 18:39:31.985945 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:31.977628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:41793 (local address 127.2.95.126:38773)
0927 18:39:31.977845 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:31.977851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:31.978186 (+ 335us) server_negotiation.cc:407] Connection header received
0927 18:39:31.978363 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:31.978368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:31.978438 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:31.978524 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:31.979331 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:31.980159 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:31.981391 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:31.981882 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:31.981984 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:31.985725 (+ 3741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:41793: BlockingRecv error: recv got EOF from 127.2.95.65:41793 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 18:39:32 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:32 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:32.433089 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:32.411889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:43663)
0927 18:39:32.412661 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:32.412677 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:32.412840 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:32.413317 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:32.413320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:32.413878 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:32.414038 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:32.414048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:32.415297 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:32.415304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:32.416546 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:32.416565 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:32.417503 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:32.417508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:32.417655 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:32.429496 (+ 11841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:32.429526 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:32.432829 (+ 3303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":512,"threads_started":1}
W20260927 18:39:32.433501 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:32.412028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:43663 (local address 127.2.95.126:38773)
0927 18:39:32.412349 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:32.412354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:32.412784 (+ 430us) server_negotiation.cc:407] Connection header received
0927 18:39:32.412987 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:32.412991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:32.413157 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:32.413257 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:32.414198 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:32.415122 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:32.416725 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:32.417327 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:32.417440 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:32.433221 (+ 15781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:43663: BlockingRecv error: recv got EOF from 127.2.95.66:43663 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
Sep 27 18:39:32 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:32 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:32.874132 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:32.865118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:53389)
0927 18:39:32.865741 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:32.865761 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:32.865938 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:32.866419 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:32.866423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:32.867021 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:32.867185 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:32.867195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:32.868241 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:32.868245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:32.869196 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:32.869210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:32.870099 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:32.870105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:32.870277 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:32.870934 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:32.870955 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:32.873807 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":353,"threads_started":1}
W20260927 18:39:32.874257 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:32.865251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:53389 (local address 127.2.95.126:38773)
0927 18:39:32.865430 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:32.865436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:32.865883 (+ 447us) server_negotiation.cc:407] Connection header received
0927 18:39:32.866101 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:32.866106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:32.866254 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:32.866375 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:32.867354 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:32.868090 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:32.869370 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:32.869927 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:32.870063 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:32.874044 (+ 3981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:53389: BlockingRecv error: recv got EOF from 127.2.95.67:53389 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 18:39:32 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:32 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:32.995832 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:32.986766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:51375)
0927 18:39:32.987438 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:32.987454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:32.987605 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:32.987924 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:32.987928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:32.988353 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:32.988511 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:32.988520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:32.989664 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:32.989702 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:32.990646 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:32.990659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:32.991607 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:32.991614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:32.991780 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:32.992360 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:32.992377 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:32.995467 (+ 3090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":428,"threads_started":1}
W20260927 18:39:32.995949 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:32.986987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:51375 (local address 127.2.95.126:38773)
0927 18:39:32.987263 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:32.987270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:32.987573 (+ 303us) server_negotiation.cc:407] Connection header received
0927 18:39:32.987728 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:32.987732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:32.987794 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:32.987874 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:32.988652 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:32.989513 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:32.990793 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:32.991396 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:32.991529 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:32.995712 (+ 4183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:51375: BlockingRecv error: recv got EOF from 127.2.95.65:51375 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 27 18:39:33 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:33 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:33.443333 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:33.434194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:60629)
0927 18:39:33.434776 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:33.434792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:33.434936 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:33.435248 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:33.435253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:33.435724 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:33.435883 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:33.435893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:33.437063 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:33.437069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:33.438185 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:33.438201 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:33.439068 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:33.439075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:33.439242 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:33.439959 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:33.439981 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:33.443074 (+ 3093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":317,"threads_started":1}
W20260927 18:39:33.443523 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:33.434397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:60629 (local address 127.2.95.126:38773)
0927 18:39:33.434632 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:33.434638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:33.434879 (+ 241us) server_negotiation.cc:407] Connection header received
0927 18:39:33.435059 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:33.435063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:33.435122 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:33.435203 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:33.436131 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:33.436897 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:33.438343 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:33.438856 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:33.439003 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:33.443312 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:60629: BlockingRecv error: recv got EOF from 127.2.95.66:60629 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 18:39:33 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:33 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:33.883577 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:33.875262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:36867)
0927 18:39:33.875895 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:33.875912 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:33.876060 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:33.876453 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:33.876459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:33.877006 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:33.877204 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:33.877219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:33.878313 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:33.878317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:33.879241 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:33.879253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:33.879986 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:33.879990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:33.880100 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:33.880725 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:33.880743 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:33.883340 (+ 2597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":358,"threads_started":1}
W20260927 18:39:33.883734 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:33.875382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:36867 (local address 127.2.95.126:38773)
0927 18:39:33.875615 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:33.875621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:33.875999 (+ 378us) server_negotiation.cc:407] Connection header received
0927 18:39:33.876177 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:33.876181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:33.876263 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:33.876372 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:33.877374 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:33.878182 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:33.879383 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:33.879845 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:33.879939 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:33.883534 (+ 3595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:36867: BlockingRecv error: recv got EOF from 127.2.95.67:36867 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 18:39:34 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:34 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:34.006398 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:33.996998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:51739)
0927 18:39:33.997544 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:33.997567 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:33.997783 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:33.998133 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:33.998138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:33.998530 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:33.998715 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:33.998725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:33.999773 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:33.999778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:34.001114 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:34.001129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:34.001967 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:34.001971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:34.002191 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:34.002790 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:34.002807 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:34.006111 (+ 3304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":269,"threads_started":1}
W20260927 18:39:34.006789 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:33.997243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:51739 (local address 127.2.95.126:38773)
0927 18:39:33.997551 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:33.997557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:33.997653 (+ 96us) server_negotiation.cc:407] Connection header received
0927 18:39:33.997928 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:33.997932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:33.997994 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:33.998074 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:33.998864 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:33.999617 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:34.001252 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:34.001806 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:34.001902 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:34.006377 (+ 4475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:51739: BlockingRecv error: recv got EOF from 127.2.95.65:51739 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
Sep 27 18:39:34 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:34 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:34.453207 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:34.444498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:45469)
0927 18:39:34.444971 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:34.444987 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:34.445137 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:34.445470 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:34.445476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:34.445999 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:34.446199 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:34.446212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:34.447302 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:34.447308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:34.448236 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:34.448249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:34.449032 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:34.449038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:34.449206 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:34.449810 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:34.449827 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:34.452959 (+ 3132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":250,"threads_started":1}
W20260927 18:39:34.453470 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:34.444613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:45469 (local address 127.2.95.126:38773)
0927 18:39:34.444826 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:34.444832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:34.445074 (+ 242us) server_negotiation.cc:407] Connection header received
0927 18:39:34.445261 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:34.445267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:34.445337 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:34.445420 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:34.446383 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:34.447149 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:34.448389 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:34.448849 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:34.448997 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:34.453153 (+ 4156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:45469: BlockingRecv error: recv got EOF from 127.2.95.66:45469 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 18:39:34 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:34 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.67@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:34.893656 11765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:34.884686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.67:58049)
0927 18:39:34.885267 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:34.885288 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:34.885421 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:34.885819 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:34.885823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:34.886283 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:34.886474 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:34.886485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:34.887661 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:34.887669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:34.888941 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:34.888956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:34.889850 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:34.889856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:34.890028 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:34.890577 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:34.890595 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:34.893362 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":339,"threads_started":1}
W20260927 18:39:34.893877 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:34.884856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:58049 (local address 127.2.95.126:38773)
0927 18:39:34.885121 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:34.885127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:34.885377 (+ 250us) server_negotiation.cc:407] Connection header received
0927 18:39:34.885529 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:34.885533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:34.885627 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:34.885758 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:34.886638 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:34.887463 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:34.889080 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:34.889689 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:34.889817 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:34.893620 (+ 3803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.67:58049: BlockingRecv error: recv got EOF from 127.2.95.67:58049 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 27 18:39:35 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:35 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.65@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:35.025363 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:35.007863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.65:58969)
0927 18:39:35.008520 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:35.008541 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:35.008684 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:35.009128 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:35.009131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:35.009515 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:35.009767 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:35.009778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.010898 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.010903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:35.011996 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:35.012016 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.012955 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.012960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:35.013128 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:35.021748 (+ 8620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:35.021779 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:35.025015 (+ 3236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":416,"threads_started":1}
W20260927 18:39:35.025604 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:35.007889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:58969 (local address 127.2.95.126:38773)
0927 18:39:35.008114 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:35.008124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:35.008663 (+ 539us) server_negotiation.cc:407] Connection header received
0927 18:39:35.008794 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:35.008801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:35.008883 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:35.008991 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:35.009955 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.010710 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.012202 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.012791 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.012914 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:35.025343 (+ 12429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.65:58969: BlockingRecv error: recv got EOF from 127.2.95.65:58969 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 27 18:39:35 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 18:39:35 dist-test-slave-9jll krb5kdc[10720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.95.66@KRBTEST.COM for oryx/127.2.95.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 18:39:35.463073 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:35.454326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:38773 (local address 127.2.95.66:39505)
0927 18:39:35.454846 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:35.454861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:35.455094 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:35.455472 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:35.455476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:35.455913 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:35.456110 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:35.456120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.457162 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.457166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:35.458148 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:35.458161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.458990 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.458996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:35.459168 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:35.459825 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:35.459843 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:35.462728 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.95.126:38773: Server oryx/127.2.95.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":296,"threads_started":1}
W20260927 18:39:35.463209 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:35.454439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:39505 (local address 127.2.95.126:38773)
0927 18:39:35.454675 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:35.454682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:35.454946 (+ 264us) server_negotiation.cc:407] Connection header received
0927 18:39:35.455230 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:35.455247 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:35.455327 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:35.455424 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:35.456253 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.457020 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.458287 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.458818 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.458969 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:35.462981 (+ 4012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.95.66:39505: BlockingRecv error: recv got EOF from 127.2.95.66:39505 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 18:39:35.738148 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 11308
I20260927 18:39:35.750825 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:35.739412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37616 (local address 127.2.95.65:32915)
0927 18:39:35.739964 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:35.739971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:35.740011 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:39:35.740074 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:35.740081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:35.740293 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:35.740459 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:35.741900 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.743055 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.744422 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.745003 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.745205 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:35.747639 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:35.747648 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:35.747663 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:35.747707 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:35.750334 (+ 2627us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.95.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 18:39:35.750600 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37616: Request ticket server kudu/127.2.95.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":278,"threads_started":1}
W20260927 18:39:35.750922 11769 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37616: Request ticket server kudu/127.2.95.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 18:39:35.751036 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 11308: Not authorized: Client connection negotiation failed: client connection to 127.2.95.65:32915: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.95.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 18:39:35.751257 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 11308
I20260927 18:39:35.765057 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 11460
I20260927 18:39:35.778098 11770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:35.766336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.2.95.66:43289)
0927 18:39:35.766959 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:35.766966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:35.766997 (+ 31us) server_negotiation.cc:407] Connection header received
0927 18:39:35.767058 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:35.767065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:35.767293 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:35.767454 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:35.768801 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.770023 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.771377 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.772087 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.772205 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:35.774854 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:35.774864 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:35.774879 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:35.774926 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:35.777528 (+ 2602us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.95.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 18:39:35.777866 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36412: Request ticket server kudu/127.2.95.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":317,"threads_started":1}
W20260927 18:39:35.778196 11770 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36412: Request ticket server kudu/127.2.95.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 18:39:35.778426 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 11460: Not authorized: Client connection negotiation failed: client connection to 127.2.95.66:43289: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.95.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 18:39:35.778545 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 11460
I20260927 18:39:35.791538 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 11591
I20260927 18:39:35.804513 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:35.792521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.2.95.67:41763)
0927 18:39:35.793088 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:35.793095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:35.793121 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:39:35.793182 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:35.793188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:35.793405 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:35.793551 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:35.795037 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.796125 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.797469 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:35.798272 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:35.798402 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:35.801234 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:35.801244 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:35.801259 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:35.801309 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:35.803942 (+ 2633us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.95.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 18:39:35.804282 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54384: Request ticket server kudu/127.2.95.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":303,"threads_started":1}
W20260927 18:39:35.804617 11771 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54384: Request ticket server kudu/127.2.95.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 18:39:35.804764 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 11591: Not authorized: Client connection negotiation failed: client connection to 127.2.95.67:41763: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.95.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 18:39:35.804924 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 11591
I20260927 18:39:35.818423 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 11239
I20260927 18:39:35.819880 11301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:35.940209 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 11239
2026-09-27T18:39:35Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21108 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T18:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:39:35Z Disabled control of system clock
I20260927 18:39:36.014833 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:40699
--webserver_interface=127.2.95.126
--webserver_port=0
--builtin_ntp_servers=127.2.95.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:40699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 18:39:36.352926 11776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:36.353237 11776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:36.353331 11776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:36.360898 11776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:39:36.361024 11776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:36.361060 11776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:36.361109 11776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:39:36.361172 11776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:39:36.370960 11776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:40699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:40699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:36.372308 11776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:36.374066 11776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:36.386350 11782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:36.386375 11781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:36.386477 11776 server_base.cc:1034] running on GCE node
W20260927 18:39:36.386608 11784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:36.387794 11776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:36.389288 11776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:36.390527 11776 hybrid_clock.cc:648] HybridClock initialized: now 1790534376390456 us; error 38 us; skew 500 ppm
I20260927 18:39:36.390873 11776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:39:36.393324 11776 webserver.cc:460] Webserver started at http://127.2.95.126:43461/ using document root <none> and password file <none>
I20260927 18:39:36.394124 11776 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:36.394233 11776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:36.394567 11776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:36.397650 11776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "75fe34e484474e7492296b73994fae5d"
format_stamp: "Formatted at 2026-09-27 18:39:36 on dist-test-slave-9jll"
server_key: "22c7b2280c4e82cfa4235fa29ad7e733"
server_key_iv: "c270e68644924c3a4d800eb959fd546f"
server_key_version: "encryptionkey@0"
I20260927 18:39:36.398597 11776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "75fe34e484474e7492296b73994fae5d"
format_stamp: "Formatted at 2026-09-27 18:39:36 on dist-test-slave-9jll"
server_key: "22c7b2280c4e82cfa4235fa29ad7e733"
server_key_iv: "c270e68644924c3a4d800eb959fd546f"
server_key_version: "encryptionkey@0"
I20260927 18:39:36.404273 11776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 18:39:36.408070 11790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:36.409651 11776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:39:36.409902 11776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "75fe34e484474e7492296b73994fae5d"
format_stamp: "Formatted at 2026-09-27 18:39:36 on dist-test-slave-9jll"
server_key: "22c7b2280c4e82cfa4235fa29ad7e733"
server_key_iv: "c270e68644924c3a4d800eb959fd546f"
server_key_version: "encryptionkey@0"
I20260927 18:39:36.410259 11776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:36.433399 11776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:36.434376 11776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:36.434669 11776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:36.452610 11776 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:40699
I20260927 18:39:36.452641 11841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:40699 every 8 connection(s)
I20260927 18:39:36.454032 11776 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:39:36.457756 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 11776
I20260927 18:39:36.457931 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:39:36.458042 11842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:36.458601 2429 external_mini_cluster.cc:1428] Setting key 08ed98022664a8e58e097588b0fdcd19
I20260927 18:39:36.470362 11842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d: Bootstrap starting.
I20260927 18:39:36.472767 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:36.460584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.2.95.126:40699)
0927 18:39:36.461601 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:36.461612 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:36.461655 (+ 43us) server_negotiation.cc:407] Connection header received
0927 18:39:36.464166 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:36.464190 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:36.464730 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:36.465106 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:36.465758 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:36.466995 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:36.468385 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:36.469109 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:36.469277 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:36.470483 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:36.470491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:36.470508 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:36.471083 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:36.471159 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:36.471185 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:36.471283 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:36.471508 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 18:39:36.471830 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":336,"threads_started":1}
I20260927 18:39:36.473866 11842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:36.474961 11842 log.cc:824] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:36.477916 11842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d: No bootstrap required, opened a new log
I20260927 18:39:36.485184 11842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fe34e484474e7492296b73994fae5d" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40699 } }
I20260927 18:39:36.485566 11842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:36.485708 11842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75fe34e484474e7492296b73994fae5d, State: Initialized, Role: FOLLOWER
I20260927 18:39:36.486294 11842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fe34e484474e7492296b73994fae5d" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40699 } }
I20260927 18:39:36.486496 11842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:39:36.486618 11842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:39:36.486779 11842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:36.489118 11842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fe34e484474e7492296b73994fae5d" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40699 } }
I20260927 18:39:36.489624 11842 leader_election.cc:302] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75fe34e484474e7492296b73994fae5d; no voters:
I20260927 18:39:36.490259 11842 leader_election.cc:288] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:39:36.490418 11847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:36.491062 11847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [term 1 LEADER]: Becoming Leader. State: Replica: 75fe34e484474e7492296b73994fae5d, State: Running, Role: LEADER
I20260927 18:39:36.491695 11847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fe34e484474e7492296b73994fae5d" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40699 } }
I20260927 18:39:36.492396 11842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:39:36.494326 11849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75fe34e484474e7492296b73994fae5d. Latest consensus state: current_term: 1 leader_uuid: "75fe34e484474e7492296b73994fae5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fe34e484474e7492296b73994fae5d" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40699 } } }
I20260927 18:39:36.494753 11849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:36.495244 11848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75fe34e484474e7492296b73994fae5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fe34e484474e7492296b73994fae5d" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40699 } } }
I20260927 18:39:36.495569 11848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:36.498680 11854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:39:36.504221 11854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:39:36.515517 11854 catalog_manager.cc:1347] Generated new cluster ID: b869db74b36440e8bcb2c43f6d05e016
I20260927 18:39:36.515657 11854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:39:36.526906 11854 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:39:36.528326 11854 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:39:36.540180 11854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75fe34e484474e7492296b73994fae5d: Generated new TSK 0
I20260927 18:39:36.541179 11854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:39:36.552481 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:40699
--builtin_ntp_servers=127.2.95.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 18:39:36.887470 11866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:36.887779 11866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:36.887897 11866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:36.895706 11866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:36.895833 11866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:36.895973 11866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:39:36.905501 11866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:40699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:36.906817 11866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:36.908638 11866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:36.922334 11874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:36.922305 11872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:36.922276 11871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:36.922997 11866 server_base.cc:1034] running on GCE node
I20260927 18:39:36.923637 11866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:36.924625 11866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:36.925819 11866 hybrid_clock.cc:648] HybridClock initialized: now 1790534376925749 us; error 57 us; skew 500 ppm
I20260927 18:39:36.926169 11866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:39:36.928632 11866 webserver.cc:460] Webserver started at http://127.2.95.65:46347/ using document root <none> and password file <none>
I20260927 18:39:36.929375 11866 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:36.929466 11866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:36.929831 11866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:36.932840 11866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "7e02cabcd9a34d78a14bdefcb1dc08c3"
format_stamp: "Formatted at 2026-09-27 18:39:36 on dist-test-slave-9jll"
server_key: "cea2e4327e731721efc6e9ebbbc64522"
server_key_iv: "d9134e90b650a0b3ea4a83015d99d100"
server_key_version: "encryptionkey@0"
I20260927 18:39:36.933827 11866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "7e02cabcd9a34d78a14bdefcb1dc08c3"
format_stamp: "Formatted at 2026-09-27 18:39:36 on dist-test-slave-9jll"
server_key: "cea2e4327e731721efc6e9ebbbc64522"
server_key_iv: "d9134e90b650a0b3ea4a83015d99d100"
server_key_version: "encryptionkey@0"
I20260927 18:39:36.939778 11866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:39:36.943746 11880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:36.945329 11866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:39:36.945535 11866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "7e02cabcd9a34d78a14bdefcb1dc08c3"
format_stamp: "Formatted at 2026-09-27 18:39:36 on dist-test-slave-9jll"
server_key: "cea2e4327e731721efc6e9ebbbc64522"
server_key_iv: "d9134e90b650a0b3ea4a83015d99d100"
server_key_version: "encryptionkey@0"
I20260927 18:39:36.945796 11866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:36.971758 11866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:36.972648 11866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:36.972908 11866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:36.973953 11866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:36.975457 11866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:36.975617 11866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:36.975728 11866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:36.975795 11866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:37.006840 11866 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:45333
I20260927 18:39:37.006903 11992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:45333 every 8 connection(s)
I20260927 18:39:37.008144 11866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:39:37.018524 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 11866
I20260927 18:39:37.018707 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:39:37.019608 2429 external_mini_cluster.cc:1428] Setting key e488ce1854593d0bc5ecc3c191ec6f08
I20260927 18:39:37.021413 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:37.010769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:55569 (local address 127.2.95.126:40699)
0927 18:39:37.011352 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:37.011357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:37.012483 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 18:39:37.014435 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:37.014440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:37.014509 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:37.014611 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:37.015856 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.016579 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.018214 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.018636 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.018746 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:37.020424 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:37.020430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:37.020435 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:37.020472 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:37.020557 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:37.020576 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:37.020658 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:37.020951 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 18:39:37.021051 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":373,"threads_started":1}
I20260927 18:39:37.022492 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:37.011264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:40699 (local address 127.2.95.65:55569)
0927 18:39:37.012276 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:37.012325 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:37.014129 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:37.014848 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:37.014856 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:37.014959 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:37.015684 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:37.015705 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.016730 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.016734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:37.018066 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:37.018077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.018875 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.018883 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:37.019189 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:37.020010 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:39:37.020013 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:37.020072 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:39:37.020190 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:37.020696 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:37.020706 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:37.020714 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:39:37.020905 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 18:39:37.021237 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":334,"threads_started":1}
I20260927 18:39:37.022734 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:40699
--builtin_ntp_servers=127.2.95.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 18:39:37.024046 11993 heartbeater.cc:347] Connected to a master server at 127.2.95.126:40699
I20260927 18:39:37.024405 11993 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:37.025310 11993 heartbeater.cc:511] Master 127.2.95.126:40699 requested a full tablet report, sending...
I20260927 18:39:37.027607 11807 ts_manager.cc:194] Registered new tserver with Master: 7e02cabcd9a34d78a14bdefcb1dc08c3 (127.2.95.65:45333)
I20260927 18:39:37.029567 11807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.95.65:55569
W20260927 18:39:37.350347 11997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:37.350644 11997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:37.350728 11997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:37.358367 11997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:37.358503 11997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:37.358613 11997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:39:37.368091 11997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:40699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:37.369359 11997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:37.371155 11997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:37.385433 12003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:37.385968 12005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:37.386021 12002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:37.391652 11997 server_base.cc:1034] running on GCE node
I20260927 18:39:37.392457 11997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:37.393699 11997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:37.394953 11997 hybrid_clock.cc:648] HybridClock initialized: now 1790534377394837 us; error 105 us; skew 500 ppm
I20260927 18:39:37.395401 11997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:39:37.398120 11997 webserver.cc:460] Webserver started at http://127.2.95.66:40915/ using document root <none> and password file <none>
I20260927 18:39:37.398876 11997 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:37.398969 11997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:37.399294 11997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:37.402649 11997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "e1d8718e5d534833956811bc1f5b757c"
format_stamp: "Formatted at 2026-09-27 18:39:37 on dist-test-slave-9jll"
server_key: "6a2e0dd1634a64a57ced9aaf41b3738c"
server_key_iv: "b89fed232e50637cf4ca55a8b8fc15f9"
server_key_version: "encryptionkey@0"
I20260927 18:39:37.403542 11997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "e1d8718e5d534833956811bc1f5b757c"
format_stamp: "Formatted at 2026-09-27 18:39:37 on dist-test-slave-9jll"
server_key: "6a2e0dd1634a64a57ced9aaf41b3738c"
server_key_iv: "b89fed232e50637cf4ca55a8b8fc15f9"
server_key_version: "encryptionkey@0"
I20260927 18:39:37.409389 11997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:39:37.413405 12011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:37.415165 11997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:39:37.415331 11997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "e1d8718e5d534833956811bc1f5b757c"
format_stamp: "Formatted at 2026-09-27 18:39:37 on dist-test-slave-9jll"
server_key: "6a2e0dd1634a64a57ced9aaf41b3738c"
server_key_iv: "b89fed232e50637cf4ca55a8b8fc15f9"
server_key_version: "encryptionkey@0"
I20260927 18:39:37.415551 11997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:37.431510 11997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:37.432551 11997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:37.432825 11997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:37.434063 11997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:37.435707 11997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:37.435796 11997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:37.435883 11997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:37.435937 11997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:37.468573 11997 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:46373
I20260927 18:39:37.468616 12123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:46373 every 8 connection(s)
I20260927 18:39:37.470049 11997 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:39:37.477955 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 11997
I20260927 18:39:37.478117 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:39:37.478664 2429 external_mini_cluster.cc:1428] Setting key 400427fb49604e8f56c7b0856b9959a6
I20260927 18:39:37.482025 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:40699
--builtin_ntp_servers=127.2.95.84:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 18:39:37.484275 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:37.472831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:47101 (local address 127.2.95.126:40699)
0927 18:39:37.473028 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:37.473033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:37.474683 (+ 1650us) server_negotiation.cc:407] Connection header received
0927 18:39:37.476533 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:37.476539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:37.476676 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:37.476765 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:37.478195 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.479220 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.481166 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.481807 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.481952 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:37.483471 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:37.483478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:37.483482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:37.483504 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:37.483541 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:37.483555 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:37.483650 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:37.483884 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 18:39:37.483985 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:39:37.485414 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:37.473380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:40699 (local address 127.2.95.66:47101)
0927 18:39:37.474495 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:37.474537 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:37.476214 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:37.477049 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:37.477059 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:37.477166 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:37.477965 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:37.477991 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.479411 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.479417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:37.480987 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:37.481001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.482037 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.482043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:37.482326 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:37.483104 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:39:37.483106 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:37.483152 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:39:37.483256 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:37.483657 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:37.483695 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:37.483701 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 18:39:37.483848 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 18:39:37.484241 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":384,"threads_started":1}
I20260927 18:39:37.487002 12124 heartbeater.cc:347] Connected to a master server at 127.2.95.126:40699
I20260927 18:39:37.487349 12124 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:37.488304 12124 heartbeater.cc:511] Master 127.2.95.126:40699 requested a full tablet report, sending...
I20260927 18:39:37.489962 11807 ts_manager.cc:194] Registered new tserver with Master: e1d8718e5d534833956811bc1f5b757c (127.2.95.66:46373)
I20260927 18:39:37.490923 11807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.95.66:47101
W20260927 18:39:37.811929 12127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:37.812248 12127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:37.812330 12127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:37.819919 12127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:37.820122 12127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:37.820266 12127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:39:37.829838 12127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:40699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:37.831156 12127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:37.832893 12127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:37.846752 12135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:37.846697 12133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:37.846720 12132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:37.847649 12127 server_base.cc:1034] running on GCE node
I20260927 18:39:37.848286 12127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:37.849311 12127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:37.850538 12127 hybrid_clock.cc:648] HybridClock initialized: now 1790534377850470 us; error 61 us; skew 500 ppm
I20260927 18:39:37.850956 12127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:39:37.853324 12127 webserver.cc:460] Webserver started at http://127.2.95.67:32945/ using document root <none> and password file <none>
I20260927 18:39:37.854104 12127 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:37.854202 12127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:37.854527 12127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:37.857502 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "40eb5b9a32e1420f905789cedcc6ac75"
format_stamp: "Formatted at 2026-09-27 18:39:37 on dist-test-slave-9jll"
server_key: "e5535c8e12628dc1c6f5229a52840576"
server_key_iv: "188c6951942b297155d2ceb8cc7a8662"
server_key_version: "encryptionkey@0"
I20260927 18:39:37.858440 12127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "40eb5b9a32e1420f905789cedcc6ac75"
format_stamp: "Formatted at 2026-09-27 18:39:37 on dist-test-slave-9jll"
server_key: "e5535c8e12628dc1c6f5229a52840576"
server_key_iv: "188c6951942b297155d2ceb8cc7a8662"
server_key_version: "encryptionkey@0"
I20260927 18:39:37.864090 12127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 18:39:37.868191 12141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:37.869860 12127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:39:37.870119 12127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "40eb5b9a32e1420f905789cedcc6ac75"
format_stamp: "Formatted at 2026-09-27 18:39:37 on dist-test-slave-9jll"
server_key: "e5535c8e12628dc1c6f5229a52840576"
server_key_iv: "188c6951942b297155d2ceb8cc7a8662"
server_key_version: "encryptionkey@0"
I20260927 18:39:37.870347 12127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:37.907752 12127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:37.908722 12127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:37.909011 12127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:37.910097 12127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:37.911693 12127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:37.911798 12127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:37.911911 12127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:37.911980 12127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:37.944876 12127 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:35361
I20260927 18:39:37.944923 12253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:35361 every 8 connection(s)
I20260927 18:39:37.946288 12127 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:39:37.948026 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 12127
I20260927 18:39:37.948155 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:39:37.948580 2429 external_mini_cluster.cc:1428] Setting key cf7976a43848a7ebecdf08b078ae2f5c
I20260927 18:39:37.959434 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:37.948824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:44727 (local address 127.2.95.126:40699)
0927 18:39:37.949031 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:37.949037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:37.950817 (+ 1780us) server_negotiation.cc:407] Connection header received
0927 18:39:37.952821 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:37.952825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:37.952894 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:37.952983 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:37.954164 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.954900 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.956441 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.956891 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.956994 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:37.958545 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:37.958550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:37.958554 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:37.958596 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:37.958630 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:37.958642 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:37.958716 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:37.959053 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 18:39:37.959155 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:39:37.960630 12256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:37.949499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:40699 (local address 127.2.95.67:44727)
0927 18:39:37.950644 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:37.950708 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:37.952548 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:37.953208 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:37.953217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:37.953312 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:37.953993 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:37.954011 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.955035 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.955039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:37.956289 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:37.956301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.957137 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.957143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:37.957365 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:37.958169 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:39:37.958173 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:37.958223 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:39:37.958319 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:37.958758 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:37.958768 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:37.958776 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:39:37.958978 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 18:39:37.959434 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":372,"threads_started":1}
I20260927 18:39:37.962327 12254 heartbeater.cc:347] Connected to a master server at 127.2.95.126:40699
I20260927 18:39:37.962687 12254 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:37.963574 12254 heartbeater.cc:511] Master 127.2.95.126:40699 requested a full tablet report, sending...
I20260927 18:39:37.965140 11807 ts_manager.cc:194] Registered new tserver with Master: 40eb5b9a32e1420f905789cedcc6ac75 (127.2.95.67:35361)
I20260927 18:39:37.966262 11807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.95.67:44727
I20260927 18:39:37.976996 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:39:37.983618 11995 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:37.981816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.2.95.126:40699)
0927 18:39:37.982006 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:37.982014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:37.982453 (+ 439us) server_negotiation.cc:407] Connection header received
0927 18:39:37.982605 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:37.982608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:37.982688 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:37.982773 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:37.983386 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59536: BlockingRecv error: recv got EOF from 127.0.0.1:59536 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:39:37.984416 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 11866
I20260927 18:39:37.992482 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:37.985342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.2.95.65:45333)
0927 18:39:37.985885 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:37.985894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:37.985924 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:39:37.985992 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:37.986001 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:37.986284 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:37.986428 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:37.986880 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.987864 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.989158 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:37.989782 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:37.989899 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:37.990966 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:37.990975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:37.990989 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:37.991599 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:37.991648 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:37.991679 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:37.991779 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:37.991977 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 18:39:37.992112 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":209,"threads_started":1}
I20260927 18:39:37.993924 11988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:38.075441 11993 heartbeater.cc:503] Master 127.2.95.126:40699 was elected leader, sending a full tablet report...
I20260927 18:39:38.076954 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 11866
W20260927 18:39:38.090394 11792 connection.cc:535] server connection from 127.2.95.65:55569 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:39:38.090683 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 11997
I20260927 18:39:38.098646 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:38.091513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54792 (local address 127.2.95.66:46373)
0927 18:39:38.092020 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:38.092028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:38.092062 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:39:38.092140 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:38.092149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:38.092394 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:38.092527 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:38.092878 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:38.093990 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:38.095205 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:38.095804 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:38.095915 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:38.096962 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:38.096970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:38.096984 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:38.097868 (+ 884us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:38.097925 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:38.097946 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:38.098042 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:38.098163 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 18:39:38.098324 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":269,"threads_started":1}
I20260927 18:39:38.100037 12119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:38.181766 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 11997
I20260927 18:39:38.194983 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 12127
I20260927 18:39:38.203649 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:38.196017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.2.95.67:35361)
0927 18:39:38.196683 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:38.196692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:38.196726 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:39:38.196788 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:38.196794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:38.197011 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:38.197159 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:38.197643 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:38.198916 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:38.200479 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:38.201049 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:38.201155 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:38.202341 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:38.202349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:38.202363 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:38.202943 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:38.202998 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:38.203017 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:38.203108 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:38.203243 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 18:39:38.203363 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":356,"threads_started":1}
I20260927 18:39:38.204871 12249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:38.286319 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 12127
I20260927 18:39:38.299142 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 11776
I20260927 18:39:38.300302 11837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:38.403474 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 11776
2026-09-27T18:39:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2456 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T18:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:39:38Z Disabled control of system clock
I20260927 18:39:38.470177 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:42701
--webserver_interface=127.2.95.126
--webserver_port=0
--builtin_ntp_servers=127.2.95.84:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:42701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:39:38.797072 12273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:38.797420 12273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:38.797504 12273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:38.805053 12273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:39:38.805189 12273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:38.805227 12273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:38.805277 12273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:39:38.805320 12273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:39:38.814934 12273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:42701
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:42701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:38.816236 12273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:38.818068 12273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:38.830730 12279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:38.830720 12281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:38.831314 12278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:38.831585 12273 server_base.cc:1034] running on GCE node
I20260927 18:39:38.832430 12273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:38.834060 12273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:38.835299 12273 hybrid_clock.cc:648] HybridClock initialized: now 1790534378835240 us; error 58 us; skew 500 ppm
I20260927 18:39:38.835691 12273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:39:38.838450 12273 webserver.cc:460] Webserver started at http://127.2.95.126:36329/ using document root <none> and password file <none>
I20260927 18:39:38.839203 12273 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:38.839329 12273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:38.839704 12273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:38.842821 12273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "b200dc0aaf74458587c2ad2f978cc020"
format_stamp: "Formatted at 2026-09-27 18:39:38 on dist-test-slave-9jll"
server_key: "cf38e4ff39dce067ffc0b12e29075e0c"
server_key_iv: "560782bde5018887d1752ffc72f49086"
server_key_version: "encryptionkey@0"
I20260927 18:39:38.843665 12273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "b200dc0aaf74458587c2ad2f978cc020"
format_stamp: "Formatted at 2026-09-27 18:39:38 on dist-test-slave-9jll"
server_key: "cf38e4ff39dce067ffc0b12e29075e0c"
server_key_iv: "560782bde5018887d1752ffc72f49086"
server_key_version: "encryptionkey@0"
I20260927 18:39:38.849115 12273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 18:39:38.852967 12287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:38.854516 12273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:39:38.854707 12273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "b200dc0aaf74458587c2ad2f978cc020"
format_stamp: "Formatted at 2026-09-27 18:39:38 on dist-test-slave-9jll"
server_key: "cf38e4ff39dce067ffc0b12e29075e0c"
server_key_iv: "560782bde5018887d1752ffc72f49086"
server_key_version: "encryptionkey@0"
I20260927 18:39:38.854926 12273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:38.860991 12273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:38.861969 12273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:38.862274 12273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:38.880362 12273 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:42701
I20260927 18:39:38.880401 12338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:42701 every 8 connection(s)
I20260927 18:39:38.881800 12273 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:39:38.885960 12339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:38.892354 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 12273
I20260927 18:39:38.892591 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:39:38.893381 2429 external_mini_cluster.cc:1428] Setting key e512ced513f6ca4dd5ea9b04032d7426
I20260927 18:39:38.898090 12339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020: Bootstrap starting.
I20260927 18:39:38.901897 12339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:38.902514 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:38.895730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.2.95.126:42701)
0927 18:39:38.896752 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:38.896764 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:38.896809 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:39:38.898572 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:38.898602 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:38.899045 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:38.899515 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:38.900701 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:38.900708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:38.900722 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:38.900745 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:38.900807 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:38.900823 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:38.900994 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:38.901116 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 18:39:38.901464 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":335,"threads_started":1}
I20260927 18:39:38.903069 12339 log.cc:824] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:38.905952 12339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020: No bootstrap required, opened a new log
I20260927 18:39:38.913518 12339 raft_consensus.cc:348] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b200dc0aaf74458587c2ad2f978cc020" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42701 } }
I20260927 18:39:38.914039 12339 raft_consensus.cc:374] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:38.914144 12339 raft_consensus.cc:729] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b200dc0aaf74458587c2ad2f978cc020, State: Initialized, Role: FOLLOWER
I20260927 18:39:38.914773 12339 consensus_queue.cc:260] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b200dc0aaf74458587c2ad2f978cc020" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42701 } }
I20260927 18:39:38.915019 12339 raft_consensus.cc:388] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:39:38.915146 12339 raft_consensus.cc:482] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:39:38.915313 12339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:38.917531 12339 raft_consensus.cc:504] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b200dc0aaf74458587c2ad2f978cc020" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42701 } }
I20260927 18:39:38.918089 12339 leader_election.cc:302] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b200dc0aaf74458587c2ad2f978cc020; no voters:
I20260927 18:39:38.918710 12339 leader_election.cc:288] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:39:38.918862 12344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:38.920090 12344 raft_consensus.cc:686] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [term 1 LEADER]: Becoming Leader. State: Replica: b200dc0aaf74458587c2ad2f978cc020, State: Running, Role: LEADER
I20260927 18:39:38.920756 12344 consensus_queue.cc:237] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b200dc0aaf74458587c2ad2f978cc020" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42701 } }
I20260927 18:39:38.921057 12339 sys_catalog.cc:564] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:39:38.923503 12346 sys_catalog.cc:455] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b200dc0aaf74458587c2ad2f978cc020. Latest consensus state: current_term: 1 leader_uuid: "b200dc0aaf74458587c2ad2f978cc020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b200dc0aaf74458587c2ad2f978cc020" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42701 } } }
I20260927 18:39:38.923929 12346 sys_catalog.cc:458] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:38.924643 12345 sys_catalog.cc:455] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b200dc0aaf74458587c2ad2f978cc020" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b200dc0aaf74458587c2ad2f978cc020" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42701 } } }
I20260927 18:39:38.925773 12345 sys_catalog.cc:458] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020 [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:38.926987 12353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:39:38.932212 12353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:39:38.941650 12353 catalog_manager.cc:1347] Generated new cluster ID: 447d9c4eb19a452584d552fb4b388411
I20260927 18:39:38.941823 12353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:39:38.967815 12353 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:39:38.968354 12353 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:39:38.984772 12353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b200dc0aaf74458587c2ad2f978cc020: Generated new TSK 0
I20260927 18:39:38.986034 12353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:39:39.001371 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42701
--builtin_ntp_servers=127.2.95.84:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 18:39:39.334954 12364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:39.335263 12364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:39.335357 12364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:39.343623 12364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:39.343763 12364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:39.343912 12364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:39:39.353330 12364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:39.354728 12364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:39.356637 12364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:39.371399 12370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:39.371604 12372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:39.371381 12369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:39.372263 12364 server_base.cc:1034] running on GCE node
I20260927 18:39:39.372915 12364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:39.373939 12364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:39.375280 12364 hybrid_clock.cc:648] HybridClock initialized: now 1790534379375214 us; error 62 us; skew 500 ppm
I20260927 18:39:39.375627 12364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:39:39.378082 12364 webserver.cc:460] Webserver started at http://127.2.95.65:42501/ using document root <none> and password file <none>
I20260927 18:39:39.378801 12364 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:39.378911 12364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:39.379266 12364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:39.382392 12364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "d75d5a189201401b8ba6341820593044"
format_stamp: "Formatted at 2026-09-27 18:39:39 on dist-test-slave-9jll"
server_key: "d59ae1b6a6f69925374d05d7b61342d1"
server_key_iv: "875c2a7a312026b9b13a30688b1b51b9"
server_key_version: "encryptionkey@0"
I20260927 18:39:39.383279 12364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "d75d5a189201401b8ba6341820593044"
format_stamp: "Formatted at 2026-09-27 18:39:39 on dist-test-slave-9jll"
server_key: "d59ae1b6a6f69925374d05d7b61342d1"
server_key_iv: "875c2a7a312026b9b13a30688b1b51b9"
server_key_version: "encryptionkey@0"
I20260927 18:39:39.388906 12364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:39:39.392859 12378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:39.394546 12364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:39:39.394732 12364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "d75d5a189201401b8ba6341820593044"
format_stamp: "Formatted at 2026-09-27 18:39:39 on dist-test-slave-9jll"
server_key: "d59ae1b6a6f69925374d05d7b61342d1"
server_key_iv: "875c2a7a312026b9b13a30688b1b51b9"
server_key_version: "encryptionkey@0"
I20260927 18:39:39.394958 12364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:39.401130 12364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:39.402040 12364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:39.402300 12364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:39.403348 12364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:39.404810 12364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:39.404903 12364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:39.405009 12364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:39.405074 12364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:39.436491 12364 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:42699
I20260927 18:39:39.436546 12490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:42699 every 8 connection(s)
I20260927 18:39:39.437844 12364 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:39:39.444166 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 12364
I20260927 18:39:39.444360 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:39:39.445163 2429 external_mini_cluster.cc:1428] Setting key ffb0cb9c8cdcb30f1d672ffd9c3968fb
I20260927 18:39:39.446579 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:39.440410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:35361 (local address 127.2.95.126:42701)
0927 18:39:39.440801 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:39.440806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:39.442052 (+ 1246us) server_negotiation.cc:407] Connection header received
0927 18:39:39.443819 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:39.443825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:39.443891 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:39.443985 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:39.445649 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:39.445653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:39.445657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:39.445762 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:39.445816 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:39.445838 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:39.445975 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:39.446291 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 18:39:39.446344 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":198,"threads_started":1}
I20260927 18:39:39.447907 12494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:39.440971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:42701 (local address 127.2.95.65:35361)
0927 18:39:39.441897 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:39.441939 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:39.443525 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:39.444248 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:39.444261 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:39.444405 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:39.445257 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:39:39.445261 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:39.445320 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:39:39.445437 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:39.445981 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:39.445991 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:39.445998 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:39:39.446299 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 18:39:39.446621 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":262,"threads_started":1}
I20260927 18:39:39.448441 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42701
--builtin_ntp_servers=127.2.95.84:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 18:39:39.449523 12491 heartbeater.cc:347] Connected to a master server at 127.2.95.126:42701
I20260927 18:39:39.449887 12491 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:39.450613 12491 heartbeater.cc:511] Master 127.2.95.126:42701 requested a full tablet report, sending...
I20260927 18:39:39.452818 12304 ts_manager.cc:194] Registered new tserver with Master: d75d5a189201401b8ba6341820593044 (127.2.95.65:42699)
W20260927 18:39:39.787889 12495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:39.788230 12495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:39.788354 12495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:39.796001 12495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:39.796149 12495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:39.796296 12495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:39:39.805835 12495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:39.807191 12495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:39.809011 12495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:39.823323 12501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:39.823294 12500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:39.823246 12503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:39.824635 12495 server_base.cc:1034] running on GCE node
I20260927 18:39:39.825383 12495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:39.826567 12495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:39.827821 12495 hybrid_clock.cc:648] HybridClock initialized: now 1790534379827758 us; error 58 us; skew 500 ppm
I20260927 18:39:39.828190 12495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:39:39.830714 12495 webserver.cc:460] Webserver started at http://127.2.95.66:42427/ using document root <none> and password file <none>
I20260927 18:39:39.831425 12495 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:39.831545 12495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:39.831874 12495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:39.834975 12495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "701358574eaf4727b0e64d1bc2aee612"
format_stamp: "Formatted at 2026-09-27 18:39:39 on dist-test-slave-9jll"
server_key: "f3b35a6d7a09816b30884837b4128935"
server_key_iv: "cb4d960492a4d17d462ae71af319d67c"
server_key_version: "encryptionkey@0"
I20260927 18:39:39.835870 12495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "701358574eaf4727b0e64d1bc2aee612"
format_stamp: "Formatted at 2026-09-27 18:39:39 on dist-test-slave-9jll"
server_key: "f3b35a6d7a09816b30884837b4128935"
server_key_iv: "cb4d960492a4d17d462ae71af319d67c"
server_key_version: "encryptionkey@0"
I20260927 18:39:39.841738 12495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:39:39.846061 12509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:39.847918 12495 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:39:39.848215 12495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "701358574eaf4727b0e64d1bc2aee612"
format_stamp: "Formatted at 2026-09-27 18:39:39 on dist-test-slave-9jll"
server_key: "f3b35a6d7a09816b30884837b4128935"
server_key_iv: "cb4d960492a4d17d462ae71af319d67c"
server_key_version: "encryptionkey@0"
I20260927 18:39:39.848482 12495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:39.855232 12495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:39.856168 12495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:39.856459 12495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:39.857479 12495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:39.859109 12495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:39.859213 12495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:39.859333 12495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:39.859404 12495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:39.893491 12495 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:45801
I20260927 18:39:39.893528 12621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:45801 every 8 connection(s)
I20260927 18:39:39.894927 12495 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:39:39.902752 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 12495
I20260927 18:39:39.902972 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:39:39.903509 2429 external_mini_cluster.cc:1428] Setting key d99970475023ab411aa2621d9e38a31f
I20260927 18:39:39.904343 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:39.897883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:52783 (local address 127.2.95.126:42701)
0927 18:39:39.898234 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:39.898238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:39.899699 (+ 1461us) server_negotiation.cc:407] Connection header received
0927 18:39:39.901501 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:39.901507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:39.901570 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:39.901752 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:39.903418 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:39.903424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:39.903427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:39.903447 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:39.903484 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:39.903496 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:39.903633 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:39.903981 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 18:39:39.904043 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260927 18:39:39.905421 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:39.898417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:42701 (local address 127.2.95.66:52783)
0927 18:39:39.899531 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:39.899580 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:39.901171 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:39.902036 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:39.902045 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:39.902160 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:39.902905 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:39:39.902908 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:39.902988 (+ 80us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:39:39.903166 (+ 178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:39.903656 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:39.903670 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:39.903678 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:39:39.904052 (+ 374us) client_negotiation.cc:240] Negotiation successful
0927 18:39:39.904376 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":361,"threads_started":1}
I20260927 18:39:39.906775 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42701
--builtin_ntp_servers=127.2.95.84:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 18:39:39.909849 12622 heartbeater.cc:347] Connected to a master server at 127.2.95.126:42701
I20260927 18:39:39.910353 12622 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:39.911015 12622 heartbeater.cc:511] Master 127.2.95.126:42701 requested a full tablet report, sending...
I20260927 18:39:39.912738 12304 ts_manager.cc:194] Registered new tserver with Master: 701358574eaf4727b0e64d1bc2aee612 (127.2.95.66:45801)
W20260927 18:39:40.256806 12625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:40.257113 12625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:40.257210 12625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:40.265146 12625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:40.265283 12625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:40.265396 12625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:39:40.275398 12625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:40.276751 12625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:40.278913 12625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:40.292681 12631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:40.292711 12633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:40.292613 12630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:40.293242 12625 server_base.cc:1034] running on GCE node
I20260927 18:39:40.293974 12625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:40.295142 12625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:40.296370 12625 hybrid_clock.cc:648] HybridClock initialized: now 1790534380296322 us; error 42 us; skew 500 ppm
I20260927 18:39:40.296713 12625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:39:40.304557 12625 webserver.cc:460] Webserver started at http://127.2.95.67:34491/ using document root <none> and password file <none>
I20260927 18:39:40.305332 12625 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:40.305430 12625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:40.305824 12625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:40.308753 12625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "892eb3dfc83c40cfbd892e49d6368123"
format_stamp: "Formatted at 2026-09-27 18:39:40 on dist-test-slave-9jll"
server_key: "f8dd22fdf9534a44fb6fb402487a135f"
server_key_iv: "9797234178a8e32d865f345bb72e29a6"
server_key_version: "encryptionkey@0"
I20260927 18:39:40.309545 12625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "892eb3dfc83c40cfbd892e49d6368123"
format_stamp: "Formatted at 2026-09-27 18:39:40 on dist-test-slave-9jll"
server_key: "f8dd22fdf9534a44fb6fb402487a135f"
server_key_iv: "9797234178a8e32d865f345bb72e29a6"
server_key_version: "encryptionkey@0"
I20260927 18:39:40.314926 12625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 18:39:40.318929 12639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:40.320533 12625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:39:40.320708 12625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "892eb3dfc83c40cfbd892e49d6368123"
format_stamp: "Formatted at 2026-09-27 18:39:40 on dist-test-slave-9jll"
server_key: "f8dd22fdf9534a44fb6fb402487a135f"
server_key_iv: "9797234178a8e32d865f345bb72e29a6"
server_key_version: "encryptionkey@0"
I20260927 18:39:40.321002 12625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:40.328447 12625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:40.329325 12625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:40.329581 12625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:40.330612 12625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:40.332118 12625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:40.332214 12625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:40.332329 12625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:40.332396 12625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:40.365952 12625 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:42161
I20260927 18:39:40.366077 12751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:42161 every 8 connection(s)
I20260927 18:39:40.367314 12625 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:39:40.371696 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 12625
I20260927 18:39:40.371888 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:39:40.372452 2429 external_mini_cluster.cc:1428] Setting key d2f708d7d379606ed1459e2862503975
I20260927 18:39:40.376434 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:40.369904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:43867 (local address 127.2.95.126:42701)
0927 18:39:40.370126 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:40.370133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:40.371914 (+ 1781us) server_negotiation.cc:407] Connection header received
0927 18:39:40.373964 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:40.373970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:40.374038 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:40.374127 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:40.375694 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:40.375699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:40.375703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:40.375720 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:40.375752 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:40.375763 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:40.375893 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:40.376093 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 18:39:40.376157 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:39:40.377332 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:40.370592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:42701 (local address 127.2.95.67:43867)
0927 18:39:40.371732 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:40.371781 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:40.373654 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:40.374420 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:40.374432 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:40.374575 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:40.375291 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:39:40.375294 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:40.375357 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:39:40.375479 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:40.375872 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:40.375882 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:40.375890 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:39:40.376173 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 18:39:40.376395 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":288,"threads_started":1}
I20260927 18:39:40.378602 12752 heartbeater.cc:347] Connected to a master server at 127.2.95.126:42701
I20260927 18:39:40.378880 12752 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:40.379454 12752 heartbeater.cc:511] Master 127.2.95.126:42701 requested a full tablet report, sending...
I20260927 18:39:40.380860 12304 ts_manager.cc:194] Registered new tserver with Master: 892eb3dfc83c40cfbd892e49d6368123 (127.2.95.67:42161)
I20260927 18:39:40.389189 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 18:39:40.395955 12493 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:39:40.394295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.2.95.126:42701)
0927 18:39:40.394584 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:40.394591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:40.394898 (+ 307us) server_negotiation.cc:407] Connection header received
0927 18:39:40.395148 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:40.395152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:40.395206 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:40.395289 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:40.395677 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42364: BlockingRecv error: recv got EOF from 127.0.0.1:42364 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 18:39:40.396821 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 12364
I20260927 18:39:40.400785 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:40.397854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.2.95.65:42699)
0927 18:39:40.398379 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:40.398387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:40.398415 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:39:40.398480 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:40.398491 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:40.398805 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:40.398941 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:40.400033 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:40.400040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:40.400053 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:40.400154 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:40.400201 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:40.400219 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:40.400336 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:40.400448 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 18:39:40.400529 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":235,"threads_started":1}
I20260927 18:39:40.402060 12486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:40.481544 12491 heartbeater.cc:503] Master 127.2.95.126:42701 was elected leader, sending a full tablet report...
I20260927 18:39:40.483496 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 12364
I20260927 18:39:40.496989 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 12495
I20260927 18:39:40.500860 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:40.497980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.2.95.66:45801)
0927 18:39:40.498475 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:40.498489 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:40.498516 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:39:40.498577 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:40.498583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:40.498806 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:40.498935 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:40.500085 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:40.500092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:40.500105 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:40.500133 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:40.500182 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:40.500198 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:40.500313 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:40.500522 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 18:39:40.500611 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
I20260927 18:39:40.502229 12617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:40.583619 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 12495
I20260927 18:39:40.597476 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 12625
I20260927 18:39:40.601454 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:40.598630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.2.95.67:42161)
0927 18:39:40.599366 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:40.599373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:40.599406 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:39:40.599469 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:40.599475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:40.599714 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:40.599863 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:40.600744 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:40.600750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:40.600762 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:39:40.600778 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:40.600816 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:39:40.600830 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:40.600961 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:40.601023 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 18:39:40.601113 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":429,"threads_started":1}
I20260927 18:39:40.602821 12747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:40.683734 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 12625
I20260927 18:39:40.696662 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 12273
I20260927 18:39:40.697913 12334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:40.796700 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 12273
2026-09-27T18:39:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2393 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:39:40 dist-test-slave-9jll krb5kdc[12770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:39:40 dist-test-slave-9jll krb5kdc[12770](info): set up 2 sockets
Sep 27 18:39:40 dist-test-slave-9jll krb5kdc[12770](info): commencing operation
krb5kdc: starting...
W20260927 18:39:41.822465 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:39:41 dist-test-slave-9jll krb5kdc[12770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:39:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:39:42.054162 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:37165
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:37165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:42.387495 12783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:42.387825 12783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:42.387930 12783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:42.395466 12783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:39:42.395591 12783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:42.395650 12783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:42.395730 12783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:39:42.395789 12783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:39:42.406091 12783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:37165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:37165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:42.407426 12783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:42.408811 12783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:42.421168 12788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:42.421216 12789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:42.421195 12791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:42.422004 12783 server_base.cc:1034] running on GCE node
I20260927 18:39:42.422814 12783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:42.424332 12783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:42.425556 12783 hybrid_clock.cc:648] HybridClock initialized: now 1790534382425502 us; error 34 us; skew 500 ppm
I20260927 18:39:42.425943 12783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:42 dist-test-slave-9jll krb5kdc[12770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534382, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:42.429138 12783 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:39:42.430866 12783 webserver.cc:460] Webserver started at http://127.2.95.126:35897/ using document root <none> and password file <none>
I20260927 18:39:42.431603 12783 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:42.431740 12783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:42.432042 12783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:42.435305 12783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "6ff3bac9f2d8469f9c124c0de0bda1a8"
format_stamp: "Formatted at 2026-09-27 18:39:42 on dist-test-slave-9jll"
server_key: "dbda66bb3761bfd93d8d0795247577f2"
server_key_iv: "9b72aee9b66b7021d18fb8ea14daa326"
server_key_version: "encryptionkey@0"
I20260927 18:39:42.436123 12783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "6ff3bac9f2d8469f9c124c0de0bda1a8"
format_stamp: "Formatted at 2026-09-27 18:39:42 on dist-test-slave-9jll"
server_key: "dbda66bb3761bfd93d8d0795247577f2"
server_key_iv: "9b72aee9b66b7021d18fb8ea14daa326"
server_key_version: "encryptionkey@0"
I20260927 18:39:42.441847 12783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:39:42.445819 12798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:42.447432 12783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:39:42.447625 12783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "6ff3bac9f2d8469f9c124c0de0bda1a8"
format_stamp: "Formatted at 2026-09-27 18:39:42 on dist-test-slave-9jll"
server_key: "dbda66bb3761bfd93d8d0795247577f2"
server_key_iv: "9b72aee9b66b7021d18fb8ea14daa326"
server_key_version: "encryptionkey@0"
I20260927 18:39:42.447835 12783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:42.462585 12783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:42.467150 12783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:42.467428 12783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:42.495033 12783 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:37165
I20260927 18:39:42.495023 12849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:37165 every 8 connection(s)
I20260927 18:39:42.496968 12783 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:39:42.501014 12850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:42.504670 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 12783
I20260927 18:39:42.504910 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:39:42.505717 2429 external_mini_cluster.cc:1428] Setting key f1f04c911d4b95f317a72dbf0e5f5dd8
I20260927 18:39:42.515501 12850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8: Bootstrap starting.
I20260927 18:39:42.519186 12850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:42.520499 12850 log.cc:824] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:39:42 dist-test-slave-9jll krb5kdc[12770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:39:42.523867 12850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8: No bootstrap required, opened a new log
I20260927 18:39:42.533500 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:42.507900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.2.95.126:37165)
0927 18:39:42.508726 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:42.508742 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:42.508793 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:39:42.511368 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:42.511404 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:42.511980 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:42.512481 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:42.514296 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:42.516354 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:42.518281 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:42.519414 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:42.519625 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:42.524557 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:42.524569 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:42.524590 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:42.524652 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:42.528624 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:42.529403 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:42.529416 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:42.529426 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:42.529656 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:42.530052 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:42.530057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:42.530061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:42.530852 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:42.531164 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:42.531540 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 18:39:42.532038 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":339,"threads_started":1}
I20260927 18:39:42.534368 12850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff3bac9f2d8469f9c124c0de0bda1a8" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37165 } }
I20260927 18:39:42.534828 12850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:42.534936 12850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff3bac9f2d8469f9c124c0de0bda1a8, State: Initialized, Role: FOLLOWER
I20260927 18:39:42.535544 12850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff3bac9f2d8469f9c124c0de0bda1a8" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37165 } }
I20260927 18:39:42.535749 12850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:39:42.535868 12850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:39:42.536028 12850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:42.538430 12850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff3bac9f2d8469f9c124c0de0bda1a8" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37165 } }
I20260927 18:39:42.538931 12850 leader_election.cc:302] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ff3bac9f2d8469f9c124c0de0bda1a8; no voters:
I20260927 18:39:42.539516 12850 leader_election.cc:288] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:39:42.539664 12855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:42.540277 12855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [term 1 LEADER]: Becoming Leader. State: Replica: 6ff3bac9f2d8469f9c124c0de0bda1a8, State: Running, Role: LEADER
I20260927 18:39:42.540830 12855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff3bac9f2d8469f9c124c0de0bda1a8" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37165 } }
I20260927 18:39:42.541518 12850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:39:42.543720 12857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ff3bac9f2d8469f9c124c0de0bda1a8. Latest consensus state: current_term: 1 leader_uuid: "6ff3bac9f2d8469f9c124c0de0bda1a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff3bac9f2d8469f9c124c0de0bda1a8" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37165 } } }
I20260927 18:39:42.543735 12856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ff3bac9f2d8469f9c124c0de0bda1a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ff3bac9f2d8469f9c124c0de0bda1a8" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37165 } } }
I20260927 18:39:42.544389 12856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:42.545724 12857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:39:42.546737 12863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:39:42.551919 12863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:39:42.562048 12863 catalog_manager.cc:1347] Generated new cluster ID: c3eea64abb484378aa65da08a125d25b
I20260927 18:39:42.562186 12863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:39:42.577008 12863 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:39:42.578416 12863 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:39:42.587153 12863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8: Generated new TSK 0
I20260927 18:39:42.588079 12863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:39:42.646661 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37165
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:43.008100 12876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:43.008423 12876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:43.008549 12876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:43.016177 12876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:43.016305 12876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:43.016446 12876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:39:43.027324 12876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:43.028685 12876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:43.030122 12876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:43.044245 12881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:43.044281 12882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:43.044322 12884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:43.044811 12876 server_base.cc:1034] running on GCE node
I20260927 18:39:43.045600 12876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:43.046905 12876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:43.048154 12876 hybrid_clock.cc:648] HybridClock initialized: now 1790534383048077 us; error 68 us; skew 500 ppm
I20260927 18:39:43.048517 12876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:43 dist-test-slave-9jll krb5kdc[12770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:43.051926 12876 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:39:43.053545 12876 webserver.cc:460] Webserver started at http://127.2.95.65:39707/ using document root <none> and password file <none>
I20260927 18:39:43.054298 12876 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:43.054412 12876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:43.054690 12876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:43.057842 12876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "de966f04ff074655a45058337b906038"
format_stamp: "Formatted at 2026-09-27 18:39:43 on dist-test-slave-9jll"
server_key: "acfc1b4d621b119ad2b33c8f1b472d09"
server_key_iv: "bd6c6808ef22c6586b0367d1f0d14f82"
server_key_version: "encryptionkey@0"
I20260927 18:39:43.058641 12876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "de966f04ff074655a45058337b906038"
format_stamp: "Formatted at 2026-09-27 18:39:43 on dist-test-slave-9jll"
server_key: "acfc1b4d621b119ad2b33c8f1b472d09"
server_key_iv: "bd6c6808ef22c6586b0367d1f0d14f82"
server_key_version: "encryptionkey@0"
I20260927 18:39:43.064266 12876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 18:39:43.068385 12891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:43.070088 12876 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:39:43.070262 12876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "de966f04ff074655a45058337b906038"
format_stamp: "Formatted at 2026-09-27 18:39:43 on dist-test-slave-9jll"
server_key: "acfc1b4d621b119ad2b33c8f1b472d09"
server_key_iv: "bd6c6808ef22c6586b0367d1f0d14f82"
server_key_version: "encryptionkey@0"
I20260927 18:39:43.070462 12876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:43.093523 12876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:43.098361 12876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:43.098647 12876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:43.099635 12876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:43.101022 12876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:43.101096 12876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:43.101207 12876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:43.101259 12876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:43.132897 12876 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:39525
I20260927 18:39:43.132942 13003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:39525 every 8 connection(s)
I20260927 18:39:43.134284 12876 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:39:43.144608 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 12876
I20260927 18:39:43.144819 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:39:43.145560 2429 external_mini_cluster.cc:1428] Setting key 86d6316748313bb0f89916a5316d0723
Sep 27 18:39:43 dist-test-slave-9jll krb5kdc[12770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:39:43.154094 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:43.136811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:45393 (local address 127.2.95.126:37165)
0927 18:39:43.137305 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:43.137310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:43.138349 (+ 1039us) server_negotiation.cc:407] Connection header received
0927 18:39:43.140131 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:43.140137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:43.140234 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:43.140325 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:43.141894 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:43.142648 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:43.144176 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:43.144557 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:43.144662 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:43.148430 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:43.148435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:43.148438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:43.148478 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:43.150896 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:43.151692 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:43.151696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:43.151698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:43.151810 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:43.152287 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:43.152291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:43.152293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:43.152581 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:43.152722 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:43.153478 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 18:39:43.153733 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":273,"threads_started":1}
I20260927 18:39:43.155081 13007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:43.137401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37165 (local address 127.2.95.65:45393)
0927 18:39:43.138151 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:43.138230 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:43.139851 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:43.140568 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:43.140577 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:43.141067 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:43.141733 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:43.141752 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:43.142809 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:43.142813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:43.144031 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:43.144042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:43.144774 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:43.144780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:43.145062 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:43.145770 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:43.145797 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:43.148217 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:43.151050 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:43.151060 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:43.151081 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:43.151573 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:43.151957 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:43.151961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:43.151963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:43.152178 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:43.152698 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:43.152707 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:43.153114 (+ 407us) client_negotiation.cc:769] Sending connection context
0927 18:39:43.153353 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 18:39:43.153936 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":310,"threads_started":1}
I20260927 18:39:43.156737 13004 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37165
I20260927 18:39:43.157027 13004 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:43.157910 13004 heartbeater.cc:511] Master 127.2.95.126:37165 requested a full tablet report, sending...
I20260927 18:39:43.160284 12815 ts_manager.cc:194] Registered new tserver with Master: de966f04ff074655a45058337b906038 (127.2.95.65:39525)
I20260927 18:39:43.162218 12815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:45393
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:39:43.196973 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37165
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:43.521334 13010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:43.521632 13010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:43.521772 13010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:43.529174 13010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:43.529299 13010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:43.529412 13010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:39:43.539456 13010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:43.540696 13010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:43.542028 13010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:43.555455 13015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:43.555447 13018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:43.555487 13016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:43.556389 13010 server_base.cc:1034] running on GCE node
I20260927 18:39:43.557040 13010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:43.558092 13010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:43.559319 13010 hybrid_clock.cc:648] HybridClock initialized: now 1790534383559272 us; error 41 us; skew 500 ppm
I20260927 18:39:43.559633 13010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:43 dist-test-slave-9jll krb5kdc[12770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:43.562781 13010 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:39:43.564314 13010 webserver.cc:460] Webserver started at http://127.2.95.66:33855/ using document root <none> and password file <none>
I20260927 18:39:43.564996 13010 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:43.565112 13010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:43.565423 13010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:43.568439 13010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "d8891f0ae60b4cd59daa30fd007c97cb"
format_stamp: "Formatted at 2026-09-27 18:39:43 on dist-test-slave-9jll"
server_key: "64b76e480c0f5aa050f664615ae1698a"
server_key_iv: "55ac2bd6061d9e0e9975783058da0d7d"
server_key_version: "encryptionkey@0"
I20260927 18:39:43.569252 13010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "d8891f0ae60b4cd59daa30fd007c97cb"
format_stamp: "Formatted at 2026-09-27 18:39:43 on dist-test-slave-9jll"
server_key: "64b76e480c0f5aa050f664615ae1698a"
server_key_iv: "55ac2bd6061d9e0e9975783058da0d7d"
server_key_version: "encryptionkey@0"
I20260927 18:39:43.574802 13010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:39:43.578495 13025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:43.579994 13010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:39:43.580178 13010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "d8891f0ae60b4cd59daa30fd007c97cb"
format_stamp: "Formatted at 2026-09-27 18:39:43 on dist-test-slave-9jll"
server_key: "64b76e480c0f5aa050f664615ae1698a"
server_key_iv: "55ac2bd6061d9e0e9975783058da0d7d"
server_key_version: "encryptionkey@0"
I20260927 18:39:43.580384 13010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:43.598224 13010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:43.603178 13010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:43.603479 13010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:43.604491 13010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:43.605911 13010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:43.605988 13010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:43.606130 13010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:43.606184 13010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:43.637219 13010 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:37947
I20260927 18:39:43.637280 13137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:37947 every 8 connection(s)
I20260927 18:39:43.638756 13010 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:39:43.639743 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 13010
I20260927 18:39:43.639942 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:39:43.640467 2429 external_mini_cluster.cc:1428] Setting key 4e9d44622625708a7adc4e4b70cb43a0
Sep 27 18:39:43 dist-test-slave-9jll krb5kdc[12770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:39:43.662007 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:43.641843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:50595 (local address 127.2.95.126:37165)
0927 18:39:43.642040 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:43.642047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:43.643569 (+ 1522us) server_negotiation.cc:407] Connection header received
0927 18:39:43.646225 (+ 2656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:43.646231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:43.646309 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:43.646401 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:43.648439 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:43.649167 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:43.651446 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:43.652077 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:43.652212 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:43.656352 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:43.656358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:43.656361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:43.656395 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:43.658808 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:43.659720 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:43.659724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:43.659726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:43.659857 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:43.660302 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:43.660305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:43.660307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:43.660555 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:43.660658 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:43.661473 (+ 815us) server_negotiation.cc:299] Negotiation successful
0927 18:39:43.661647 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:39:43.663473 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:43.642465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37165 (local address 127.2.95.66:50595)
0927 18:39:43.643384 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:43.643443 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:43.645854 (+ 2411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:43.646678 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:43.646690 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:43.647398 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:43.648211 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:43.648239 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:43.649359 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:43.649365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:43.651285 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:43.651301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:43.652340 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:43.652346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:43.652680 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:43.653377 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:43.653405 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:43.656078 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:43.658974 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:43.658984 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:43.659003 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:43.659572 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:43.659955 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:43.659959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:43.659962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:43.660207 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:43.660665 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:43.660673 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:43.661117 (+ 444us) client_negotiation.cc:769] Sending connection context
0927 18:39:43.661408 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 18:39:43.662022 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":341,"threads_started":1}
I20260927 18:39:43.665102 13138 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37165
I20260927 18:39:43.665452 13138 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
I20260927 18:39:43.666519 13138 heartbeater.cc:511] Master 127.2.95.126:37165 requested a full tablet report, sending...
I20260927 18:39:43.668238 12815 ts_manager.cc:194] Registered new tserver with Master: d8891f0ae60b4cd59daa30fd007c97cb (127.2.95.66:37947)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
I20260927 18:39:43.669278 12815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:50595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:39:43.697849 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37165
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:39:44.025007 13143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:39:44.025310 13143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:39:44.025405 13143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:39:44.032799 13143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:39:44.032927 13143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:39:44.033051 13143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:39:44.043215 13143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:39:44.044472 13143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:39:44.045809 13143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:39:44.058774 13149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:44.058804 13151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:39:44.058738 13148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:39:44.060058 13143 server_base.cc:1034] running on GCE node
I20260927 18:39:44.060753 13143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:39:44.061744 13143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:39:44.062963 13143 hybrid_clock.cc:648] HybridClock initialized: now 1790534384062909 us; error 51 us; skew 500 ppm
I20260927 18:39:44.063341 13143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:39:44 dist-test-slave-9jll krb5kdc[12770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:39:44.066856 13143 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:39:44.068532 13143 webserver.cc:460] Webserver started at http://127.2.95.67:46165/ using document root <none> and password file <none>
I20260927 18:39:44.069226 13143 fs_manager.cc:359] Metadata directory not provided
I20260927 18:39:44.069346 13143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:39:44.069765 13143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:39:44.072750 13143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "90f397f0f990407a9fc2dacb5449cbf4"
format_stamp: "Formatted at 2026-09-27 18:39:44 on dist-test-slave-9jll"
server_key: "f62d7002572ed3b2bf5904af30b782c7"
server_key_iv: "b62e4a5c3f98c3afc038777cfc5db584"
server_key_version: "encryptionkey@0"
I20260927 18:39:44.073557 13143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "90f397f0f990407a9fc2dacb5449cbf4"
format_stamp: "Formatted at 2026-09-27 18:39:44 on dist-test-slave-9jll"
server_key: "f62d7002572ed3b2bf5904af30b782c7"
server_key_iv: "b62e4a5c3f98c3afc038777cfc5db584"
server_key_version: "encryptionkey@0"
I20260927 18:39:44.078956 13143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 18:39:44.082690 13158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:44.084182 13143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:39:44.084336 13143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "90f397f0f990407a9fc2dacb5449cbf4"
format_stamp: "Formatted at 2026-09-27 18:39:44 on dist-test-slave-9jll"
server_key: "f62d7002572ed3b2bf5904af30b782c7"
server_key_iv: "b62e4a5c3f98c3afc038777cfc5db584"
server_key_version: "encryptionkey@0"
I20260927 18:39:44.084527 13143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:39:44.112361 13143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:39:44.116482 13143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:39:44.116756 13143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:39:44.117724 13143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:39:44.119076 13143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:39:44.119163 13143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:44.119279 13143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:39:44.119329 13143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:39:44.150627 13143 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:41019
I20260927 18:39:44.150658 13270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:41019 every 8 connection(s)
I20260927 18:39:44.151940 13143 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:39:44.160409 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 13143
I20260927 18:39:44.160626 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:39:44.161144 2429 external_mini_cluster.cc:1428] Setting key dc075a287d04f99895732e851a9da8ed
I20260927 18:39:44.165380 13004 heartbeater.cc:503] Master 127.2.95.126:37165 was elected leader, sending a full tablet report...
Sep 27 18:39:44 dist-test-slave-9jll krb5kdc[12770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:39:44.172029 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.154497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:57209 (local address 127.2.95.126:37165)
0927 18:39:44.154652 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:44.154656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:44.155837 (+ 1181us) server_negotiation.cc:407] Connection header received
0927 18:39:44.157587 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:44.157591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:44.157714 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:44.157809 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:44.159415 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.160180 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.162232 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.162753 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.162865 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.166701 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:44.166707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:44.166711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:44.166749 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:44.169160 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:44.169908 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:44.169912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:44.169914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:44.170136 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:44.170517 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:44.170520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:44.170522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:44.170800 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:44.170906 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:44.171598 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 18:39:44.171768 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:39:44.172964 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.154948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37165 (local address 127.2.95.67:57209)
0927 18:39:44.155638 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:44.155681 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:44.157328 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:44.158012 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:44.158021 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:44.158549 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:39:44.159221 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:44.159240 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.160366 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.160372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:44.162092 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:44.162112 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.162989 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.162996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:44.163305 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.164083 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:39:44.164114 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:39:44.166499 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:39:44.169295 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:44.169302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:44.169315 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:44.169786 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:44.170238 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:39:44.170241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:39:44.170243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:39:44.170419 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:39:44.170892 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:39:44.170898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:39:44.171218 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 18:39:44.171442 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 18:39:44.171845 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":286,"threads_started":1}
I20260927 18:39:44.174417 13271 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37165
I20260927 18:39:44.174727 13271 heartbeater.cc:464] Registering TS with master...
I20260927 18:39:44.175589 13271 heartbeater.cc:511] Master 127.2.95.126:37165 requested a full tablet report, sending...
I20260927 18:39:44.176934 12815 ts_manager.cc:194] Registered new tserver with Master: 90f397f0f990407a9fc2dacb5449cbf4 (127.2.95.67:41019)
I20260927 18:39:44.177768 12815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:57209
I20260927 18:39:44.189469 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:39:44.206977 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.194404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.2.95.126:37165)
0927 18:39:44.194705 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:44.194712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:44.195000 (+ 288us) server_negotiation.cc:407] Connection header received
0927 18:39:44.195240 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:44.195244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:44.195309 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:44.195394 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:44.196789 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.197510 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.198864 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.199300 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.199428 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.201786 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:44.201791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:44.201794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:44.201831 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:44.204264 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:44.204954 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:44.204959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:44.204962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:44.205118 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:44.205485 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:44.205489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:44.205491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:44.205858 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:44.206088 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:44.206424 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 18:39:44.206646 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 18:39:44.211637 12815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:39:44.218446 12815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:39:44.253122 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.231902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:41019 (local address 127.0.0.1:41654)
0927 18:39:44.244803 (+ 12901us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:44.244830 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:44.245049 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:44.246524 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:44.246529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:44.246558 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:44.247128 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:44.247158 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.248562 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.248567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:44.250653 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:44.250664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.252162 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.252167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:44.252311 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.252360 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 18:39:44.252498 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 18:39:44.252722 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12729,"mutex_wait_us":25,"thread_start_us":13258,"threads_started":1}
I20260927 18:39:44.253894 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.232208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.2.95.67:41019)
0927 18:39:44.244620 (+ 12412us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:44.244631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:44.244955 (+ 324us) server_negotiation.cc:407] Connection header received
0927 18:39:44.245856 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:44.245866 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:44.246227 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:44.246431 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:44.247357 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.248431 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.250805 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.251966 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.252090 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.252325 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:44.253355 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0927 18:39:44.253527 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12180,"thread_start_us":12451,"threads_started":1}
I20260927 18:39:44.254765 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.232355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39525 (local address 127.0.0.1:53946)
0927 18:39:44.245454 (+ 13099us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:44.245475 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:44.245613 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:44.246686 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:44.246690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:44.246718 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:44.247263 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:44.247276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.249176 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.249181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:44.251756 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:44.251766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.253913 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.253919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:44.254079 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.254112 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:39:44.254208 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 18:39:44.254339 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12948,"mutex_wait_us":123,"thread_start_us":12816,"threads_started":1}
I20260927 18:39:44.255577 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.230602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:37947 (local address 127.0.0.1:43700)
0927 18:39:44.231341 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:44.231383 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:44.231551 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:44.246488 (+ 14937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:44.246497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:44.246529 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:44.247211 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:44.247225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.249176 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.249181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:44.251732 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:44.251747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.254736 (+ 2989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.254743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:44.254987 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.255020 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:39:44.255079 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 18:39:44.255205 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":99,"spinlock_wait_cycles":15104,"thread_start_us":324,"threads_started":1}
I20260927 18:39:44.255633 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.232996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.2.95.65:39525)
0927 18:39:44.240130 (+ 7134us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:44.240140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:44.245857 (+ 5717us) server_negotiation.cc:407] Connection header received
0927 18:39:44.246017 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:44.246024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:44.246229 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:44.246431 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:44.247459 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.249016 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.251887 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.253411 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.253532 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.253772 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:44.255074 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0927 18:39:44.255323 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6933,"thread_start_us":7020,"threads_started":1}
I20260927 18:39:44.256526 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.230785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.2.95.66:37947)
0927 18:39:44.245843 (+ 15058us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:44.245851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:44.245874 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:39:44.245981 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:44.245990 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:44.246225 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:44.246938 (+ 713us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:44.247380 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.249016 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.252422 (+ 3406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.254501 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.255815 (+ 1314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.256004 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:44.256077 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 18:39:44.256229 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14922,"thread_start_us":14742,"threads_started":1}
I20260927 18:39:44.257400 13206 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6 (DEFAULT_TABLE table=test-table [id=32946d265a8b4293a7fc8ccb40480de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:44.259001 13206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:44.258910 12939 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6 (DEFAULT_TABLE table=test-table [id=32946d265a8b4293a7fc8ccb40480de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:44.260288 12939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:44.259905 13073 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6 (DEFAULT_TABLE table=test-table [id=32946d265a8b4293a7fc8ccb40480de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:39:44.261157 13073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:39:44.271023 13289 tablet_bootstrap.cc:492] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038: Bootstrap starting.
I20260927 18:39:44.274716 13289 tablet_bootstrap.cc:654] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:44.275863 13289 log.cc:824] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:44.278775 13290 tablet_bootstrap.cc:492] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4: Bootstrap starting.
I20260927 18:39:44.279698 13291 tablet_bootstrap.cc:492] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb: Bootstrap starting.
I20260927 18:39:44.280645 13289 tablet_bootstrap.cc:492] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038: No bootstrap required, opened a new log
I20260927 18:39:44.280918 13289 ts_tablet_manager.cc:1397] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 18:39:44.282419 13290 tablet_bootstrap.cc:654] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:44.283167 13291 tablet_bootstrap.cc:654] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb: Neither blocks nor log segments found. Creating new log.
I20260927 18:39:44.283682 13290 log.cc:824] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:44.284245 13291 log.cc:824] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:39:44.286999 13290 tablet_bootstrap.cc:492] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4: No bootstrap required, opened a new log
I20260927 18:39:44.286998 13291 tablet_bootstrap.cc:492] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb: No bootstrap required, opened a new log
I20260927 18:39:44.287290 13290 ts_tablet_manager.cc:1397] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 18:39:44.287290 13291 ts_tablet_manager.cc:1397] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 18:39:44.289565 13289 raft_consensus.cc:348] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891f0ae60b4cd59daa30fd007c97cb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37947 } } peers { permanent_uuid: "90f397f0f990407a9fc2dacb5449cbf4" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41019 } } peers { permanent_uuid: "de966f04ff074655a45058337b906038" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39525 } }
I20260927 18:39:44.289987 13289 raft_consensus.cc:374] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:44.290079 13289 raft_consensus.cc:729] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de966f04ff074655a45058337b906038, State: Initialized, Role: FOLLOWER
I20260927 18:39:44.290725 13289 consensus_queue.cc:260] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891f0ae60b4cd59daa30fd007c97cb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37947 } } peers { permanent_uuid: "90f397f0f990407a9fc2dacb5449cbf4" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41019 } } peers { permanent_uuid: "de966f04ff074655a45058337b906038" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39525 } }
I20260927 18:39:44.292486 13289 ts_tablet_manager.cc:1428] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 18:39:44.297312 13291 raft_consensus.cc:348] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891f0ae60b4cd59daa30fd007c97cb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37947 } } peers { permanent_uuid: "90f397f0f990407a9fc2dacb5449cbf4" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41019 } } peers { permanent_uuid: "de966f04ff074655a45058337b906038" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39525 } }
I20260927 18:39:44.297767 13291 raft_consensus.cc:374] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:44.297879 13291 raft_consensus.cc:729] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8891f0ae60b4cd59daa30fd007c97cb, State: Initialized, Role: FOLLOWER
I20260927 18:39:44.298439 13291 consensus_queue.cc:260] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891f0ae60b4cd59daa30fd007c97cb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37947 } } peers { permanent_uuid: "90f397f0f990407a9fc2dacb5449cbf4" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41019 } } peers { permanent_uuid: "de966f04ff074655a45058337b906038" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39525 } }
I20260927 18:39:44.299161 13290 raft_consensus.cc:348] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891f0ae60b4cd59daa30fd007c97cb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37947 } } peers { permanent_uuid: "90f397f0f990407a9fc2dacb5449cbf4" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41019 } } peers { permanent_uuid: "de966f04ff074655a45058337b906038" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39525 } }
I20260927 18:39:44.299489 13138 heartbeater.cc:503] Master 127.2.95.126:37165 was elected leader, sending a full tablet report...
I20260927 18:39:44.299569 13290 raft_consensus.cc:374] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:39:44.299669 13290 raft_consensus.cc:729] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f397f0f990407a9fc2dacb5449cbf4, State: Initialized, Role: FOLLOWER
I20260927 18:39:44.300097 13291 ts_tablet_manager.cc:1428] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260927 18:39:44.300268 13290 consensus_queue.cc:260] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891f0ae60b4cd59daa30fd007c97cb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37947 } } peers { permanent_uuid: "90f397f0f990407a9fc2dacb5449cbf4" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41019 } } peers { permanent_uuid: "de966f04ff074655a45058337b906038" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39525 } }
I20260927 18:39:44.301313 13271 heartbeater.cc:503] Master 127.2.95.126:37165 was elected leader, sending a full tablet report...
I20260927 18:39:44.301846 13290 ts_tablet_manager.cc:1428] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260927 18:39:44.335008 13295 raft_consensus.cc:482] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:39:44.335300 13295 raft_consensus.cc:504] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891f0ae60b4cd59daa30fd007c97cb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37947 } } peers { permanent_uuid: "90f397f0f990407a9fc2dacb5449cbf4" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41019 } } peers { permanent_uuid: "de966f04ff074655a45058337b906038" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39525 } }
I20260927 18:39:44.336930 13295 leader_election.cc:288] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8891f0ae60b4cd59daa30fd007c97cb (127.2.95.66:37947), 90f397f0f990407a9fc2dacb5449cbf4 (127.2.95.67:41019)
I20260927 18:39:44.346374 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.337415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:34117 (local address 127.2.95.66:37947)
0927 18:39:44.337612 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:44.337618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:44.338413 (+ 795us) server_negotiation.cc:407] Connection header received
0927 18:39:44.338648 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:44.338654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:44.338722 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:44.338803 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:44.339550 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.340673 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.343900 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.345144 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.345308 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.345417 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:44.345958 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 18:39:44.346102 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:39:44.346405 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.337596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:36235 (local address 127.2.95.67:41019)
0927 18:39:44.337783 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:44.337789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:44.338474 (+ 685us) server_negotiation.cc:407] Connection header received
0927 18:39:44.338605 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:44.338611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:44.338690 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:44.338800 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:39:44.339542 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.340650 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.343900 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.345213 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.345322 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.345415 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:44.345945 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 18:39:44.346177 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 18:39:44.346554 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.337259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:37947 (local address 127.2.95.65:34117)
0927 18:39:44.338262 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:44.338283 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:44.338463 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:44.338827 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:44.338832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:44.338871 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:44.339379 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:44.339388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.340823 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.340829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:44.343674 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:44.343696 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.345428 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.345436 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:44.345732 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.345773 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:39:44.345930 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 18:39:44.346038 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":186,"spinlock_wait_cycles":3840,"thread_start_us":768,"threads_started":1}
I20260927 18:39:44.346923 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.337305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:41019 (local address 127.2.95.65:36235)
0927 18:39:44.338301 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0927 18:39:44.338321 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:39:44.338469 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:39:44.338827 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:39:44.338832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:39:44.338863 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:39:44.339375 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:44.339387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.340824 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.340829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:44.343674 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:39:44.343696 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.345441 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.345447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:39:44.345691 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.345770 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 18:39:44.345874 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 18:39:44.346028 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":254,"spinlock_wait_cycles":49792,"thread_start_us":557,"threads_started":1}
I20260927 18:39:44.347461 13093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb6431b8d3d4589ac3c0b3ee913e9a6" candidate_uuid: "de966f04ff074655a45058337b906038" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8891f0ae60b4cd59daa30fd007c97cb" is_pre_election: true
I20260927 18:39:44.347946 13093 raft_consensus.cc:2454] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de966f04ff074655a45058337b906038 in term 0.
I20260927 18:39:44.348641 12893 leader_election.cc:302] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8891f0ae60b4cd59daa30fd007c97cb, de966f04ff074655a45058337b906038; no voters:
I20260927 18:39:44.349059 13295 raft_consensus.cc:2781] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:39:44.349195 13295 raft_consensus.cc:482] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:39:44.349285 13295 raft_consensus.cc:3037] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:44.349157 13226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb6431b8d3d4589ac3c0b3ee913e9a6" candidate_uuid: "de966f04ff074655a45058337b906038" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f397f0f990407a9fc2dacb5449cbf4" is_pre_election: true
I20260927 18:39:44.349718 13226 raft_consensus.cc:2454] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de966f04ff074655a45058337b906038 in term 0.
I20260927 18:39:44.351766 13295 raft_consensus.cc:504] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891f0ae60b4cd59daa30fd007c97cb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37947 } } peers { permanent_uuid: "90f397f0f990407a9fc2dacb5449cbf4" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41019 } } peers { permanent_uuid: "de966f04ff074655a45058337b906038" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39525 } }
I20260927 18:39:44.352383 13295 leader_election.cc:288] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [CANDIDATE]: Term 1 election: Requested vote from peers d8891f0ae60b4cd59daa30fd007c97cb (127.2.95.66:37947), 90f397f0f990407a9fc2dacb5449cbf4 (127.2.95.67:41019)
I20260927 18:39:44.352898 13226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb6431b8d3d4589ac3c0b3ee913e9a6" candidate_uuid: "de966f04ff074655a45058337b906038" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f397f0f990407a9fc2dacb5449cbf4"
I20260927 18:39:44.352901 13093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb6431b8d3d4589ac3c0b3ee913e9a6" candidate_uuid: "de966f04ff074655a45058337b906038" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8891f0ae60b4cd59daa30fd007c97cb"
I20260927 18:39:44.353086 13226 raft_consensus.cc:3037] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:44.353087 13093 raft_consensus.cc:3037] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:39:44.355311 13226 raft_consensus.cc:2454] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de966f04ff074655a45058337b906038 in term 1.
I20260927 18:39:44.355393 13093 raft_consensus.cc:2454] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de966f04ff074655a45058337b906038 in term 1.
I20260927 18:39:44.355772 12894 leader_election.cc:302] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90f397f0f990407a9fc2dacb5449cbf4, de966f04ff074655a45058337b906038; no voters:
I20260927 18:39:44.356081 13295 raft_consensus.cc:2781] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:39:44.356551 13295 raft_consensus.cc:686] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 1 LEADER]: Becoming Leader. State: Replica: de966f04ff074655a45058337b906038, State: Running, Role: LEADER
I20260927 18:39:44.357017 13295 consensus_queue.cc:237] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891f0ae60b4cd59daa30fd007c97cb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37947 } } peers { permanent_uuid: "90f397f0f990407a9fc2dacb5449cbf4" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41019 } } peers { permanent_uuid: "de966f04ff074655a45058337b906038" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39525 } }
I20260927 18:39:44.362023 12815 catalog_manager.cc:5495] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 reported cstate change: term changed from 0 to 1, leader changed from <none> to de966f04ff074655a45058337b906038 (127.2.95.65). New cstate: current_term: 1 leader_uuid: "de966f04ff074655a45058337b906038" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8891f0ae60b4cd59daa30fd007c97cb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f397f0f990407a9fc2dacb5449cbf4" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de966f04ff074655a45058337b906038" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39525 } health_report { overall_health: HEALTHY } } }
I20260927 18:39:44.378805 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.373644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.2.95.65:39525)
0927 18:39:44.373899 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:44.373905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:44.373926 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:39:44.374025 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:44.374028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:44.374102 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:44.374230 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:39:44.374593 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.375310 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.376630 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.377037 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.377142 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.377700 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:39:44.378222 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:39:44.378367 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:44.378463 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 18:39:44.378547 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260927 18:39:44.386656 13005 tablet.cc:2369] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:39:44.387480 13226 raft_consensus.cc:1261] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4 [term 1 FOLLOWER]: Refusing update from remote peer de966f04ff074655a45058337b906038: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:44.387514 13093 raft_consensus.cc:1261] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb [term 1 FOLLOWER]: Refusing update from remote peer de966f04ff074655a45058337b906038: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:39:44.388443 13300 consensus_queue.cc:1035] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8891f0ae60b4cd59daa30fd007c97cb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:39:44.388772 13295 consensus_queue.cc:1035] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f397f0f990407a9fc2dacb5449cbf4" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:39:44.391016 13139 tablet.cc:2369] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:39:44.415498 12814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:44.415853 12814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:39:44.420014 12814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ff3bac9f2d8469f9c124c0de0bda1a8: Sending DeleteTablet for 3 replicas of tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6
I20260927 18:39:44.421208 13206 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:44 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:41654
I20260927 18:39:44.421238 13073 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:44 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:43700
I20260927 18:39:44.421332 12939 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:39:44 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:53946
I20260927 18:39:44.423614 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 12876
I20260927 18:39:44.434319 13315 tablet_replica.cc:330] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038: stopping tablet replica
I20260927 18:39:44.434862 13315 raft_consensus.cc:2229] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:39:44.435724 13315 raft_consensus.cc:2258] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:44.438073 13315 ts_tablet_manager.cc:1905] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:44.446456 13315 ts_tablet_manager.cc:1918] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:44.446671 13315 log.cc:1196] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/9bb6431b8d3d4589ac3c0b3ee913e9a6
I20260927 18:39:44.447237 13315 ts_tablet_manager.cc:1939] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P de966f04ff074655a45058337b906038: Deleting consensus metadata
I20260927 18:39:44.449142 12800 catalog_manager.cc:4841] TS de966f04ff074655a45058337b906038 (127.2.95.65:39525): tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6 (table test-table [id=32946d265a8b4293a7fc8ccb40480de5]) successfully deleted
I20260927 18:39:44.450027 13313 tablet_replica.cc:330] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4: stopping tablet replica
I20260927 18:39:44.450488 13313 raft_consensus.cc:2229] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:44.451013 13313 raft_consensus.cc:2258] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:44.452365 13314 tablet_replica.cc:330] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb: stopping tablet replica
I20260927 18:39:44.452834 13314 raft_consensus.cc:2229] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:39:44.453291 13314 raft_consensus.cc:2258] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:39:44.453606 13313 ts_tablet_manager.cc:1905] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:39:44.455412 13314 ts_tablet_manager.cc:1905] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 18:39:44 dist-test-slave-9jll krb5kdc[12770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:39:44.460646 13313 ts_tablet_manager.cc:1918] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:44.460929 13313 log.cc:1196] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/9bb6431b8d3d4589ac3c0b3ee913e9a6
I20260927 18:39:44.461513 13313 ts_tablet_manager.cc:1939] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P 90f397f0f990407a9fc2dacb5449cbf4: Deleting consensus metadata
I20260927 18:39:44.462606 13314 ts_tablet_manager.cc:1918] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:39:44.462778 13314 log.cc:1196] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/9bb6431b8d3d4589ac3c0b3ee913e9a6
I20260927 18:39:44.463286 13314 ts_tablet_manager.cc:1939] T 9bb6431b8d3d4589ac3c0b3ee913e9a6 P d8891f0ae60b4cd59daa30fd007c97cb: Deleting consensus metadata
I20260927 18:39:44.464676 12801 catalog_manager.cc:4841] TS 90f397f0f990407a9fc2dacb5449cbf4 (127.2.95.67:41019): tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6 (table test-table [id=32946d265a8b4293a7fc8ccb40480de5]) successfully deleted
I20260927 18:39:44.464941 12800 catalog_manager.cc:4841] TS d8891f0ae60b4cd59daa30fd007c97cb (127.2.95.66:37947): tablet 9bb6431b8d3d4589ac3c0b3ee913e9a6 (table test-table [id=32946d265a8b4293a7fc8ccb40480de5]) successfully deleted
I20260927 18:39:44.467453 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.425313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.2.95.65:39525)
0927 18:39:44.425531 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:44.425538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:44.448035 (+ 22497us) server_negotiation.cc:407] Connection header received
0927 18:39:44.448619 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:44.448626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:44.448725 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:44.448800 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:44.451076 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.452258 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.454322 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.455044 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.455149 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.460463 (+ 5314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:44.460483 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:44.460503 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:44.460565 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:44.464307 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:44.465340 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:44.465347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:44.465350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:44.465496 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:44.465929 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:44.465933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:44.465935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:44.466293 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:44.466567 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:44.466891 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 18:39:44.467133 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 18:39:44.468146 12999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:44.581130 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 12876
I20260927 18:39:44.596537 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 13010
Sep 27 18:39:44 dist-test-slave-9jll krb5kdc[12770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:39:44.611768 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.597488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.2.95.66:37947)
0927 18:39:44.597649 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:44.597653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:44.597690 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:39:44.597903 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:44.597909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:44.597993 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:44.598170 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:44.599379 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.600141 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.601359 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.601839 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.601953 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.606254 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:44.606271 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:44.606288 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:44.606329 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:44.609113 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:44.609919 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:44.609927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:44.609929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:44.610085 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:44.610396 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:44.610399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:44.610401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:44.610795 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:44.611003 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:44.611298 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 18:39:44.611463 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:39:44.612389 13133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:44.700446 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 13010
I20260927 18:39:44.714908 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 13143
Sep 27 18:39:44 dist-test-slave-9jll krb5kdc[12770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:39:44.730783 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:39:44.715791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.2.95.67:41019)
0927 18:39:44.715947 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 18:39:44.715952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:39:44.715969 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:39:44.716024 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:39:44.716028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:39:44.716098 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:39:44.716187 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:39:44.717461 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.718385 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.719813 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:39:44.720326 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:39:44.720432 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:39:44.725207 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:39:44.725223 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:39:44.725240 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:39:44.725285 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:39:44.727967 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:44.728666 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:44.728675 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:44.728677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:44.728811 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:39:44.729259 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:39:44.729264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:39:44.729266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:39:44.729704 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:39:44.729954 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 18:39:44.730285 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 18:39:44.730453 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 18:39:44.731395 13266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:44.819376 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 13143
I20260927 18:39:44.832943 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 12783
I20260927 18:39:44.834079 12845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:39:44.939800 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 12783
2026-09-27T18:39:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4144 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:39:45 dist-test-slave-9jll krb5kdc[13323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:39:45 dist-test-slave-9jll krb5kdc[13323](info): set up 2 sockets
Sep 27 18:39:45 dist-test-slave-9jll krb5kdc[13323](info): commencing operation
krb5kdc: starting...
W20260927 18:39:45.966492 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:39:46 dist-test-slave-9jll krb5kdc[13323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:39:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:39:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.95.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.95.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/rangeradmin_127.2.95.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/rangeradmin_127.2.95.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.95.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.95.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/rangerlookup_127.2.95.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/rangerlookup_127.2.95.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.95.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.95.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/HTTP_127.2.95.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/HTTP_127.2.95.84@KRBTEST.COM.keytab.
I20260927 18:39:46.305994 2429 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknccDtC/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/postgres -l logfile start
2026-09-27 18:39:49.034 UTC [13353] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 18:39:49.034 UTC [13353] LOG: listening on IPv4 address "127.2.95.84", port 49719
2026-09-27 18:39:49.038 UTC [13353] LOG: listening on Unix socket "/tmp/.s.PGSQL.49719"
2026-09-27 18:39:49.062 UTC [13356] LOG: database system was shut down at 2026-09-27 18:39:47 UTC
2026-09-27 18:39:49.066 UTC [13353] LOG: database system is ready to accept connections
I20260927 18:39:49.966056 2429 mini_postgres.cc:96] Postgres bound to 49719
2026-09-27 18:39:49.982 UTC [13364] FATAL: database "slave" does not exist
127.2.95.84:49719 - accepting connections
I20260927 18:39:49.983618 2429 mini_ranger.cc:161] Starting Ranger...
I20260927 18:39:50.013291 2429 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 18:39:50.603224 2429 mini_ranger.cc:87] Created ranger Postgres database
I20260927 18:39:50.603350 2429 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-admin
2026-09-27 18:39:50,891 [I] DB FLAVOR :POSTGRES
2026-09-27 18:39:50,892 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:39:50,892 [I] Checking connection..
2026-09-27 18:39:50,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 18:39:51,152 [I] Checking connection passed.
2026-09-27 18:39:51,153 [I] --------- Verifying version history table ---------
2026-09-27 18:39:51,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:39:51,419 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:39:51,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:39:51,684 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:39:51,684 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 18:39:51,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 18:39:51.950 UTC [13428] WARNING: there is no transaction in progress
2026-09-27 18:39:51,956 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 18:39:51,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:39:52,210 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 18:39:52,210 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:39:52,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 18:39:52,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 18:39:52,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9jll.c.gcp-upstream.internal','N') ;"
2026-09-27 18:39:52,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:39:53,235 [I] Table x_portal_user does not exist in database ranger
2026-09-27 18:39:53,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:39:53,506 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 18:39:53,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:39:53,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:39:54,027 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 18:39:54,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 18:39:55.117 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.136 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.156 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.174 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.195 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.218 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.276 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.284 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.299 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.310 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.321 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.338 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.349 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.358 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.368 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:55.377 UTC [13536] WARNING: there is no transaction in progress
2026-09-27 18:39:56,178 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 18:39:56,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 18:39:56,437 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 18:39:56,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:39:56,700 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 18:39:56,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:39:56,970 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 18:39:56,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:39:57,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:39:57,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9jll.c.gcp-upstream.internal';"
2026-09-27 18:39:57,703 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 18:39:57,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 18:39:57,962 [I] DB_PATCHES have already been applied
I20260927 18:39:57.968683 2429 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasknccDtC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 18:39:57.968776 2429 mini_ranger.cc:193] Using host: 127.2.95.84
I20260927 18:39:57.982874 2429 mini_ranger.cc:227] Ranger admin URL: http://127.2.95.84:60727
Sep 27, 2026 6:39:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:39:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:39:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:39:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:39:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:39:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:39:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 18:39:59.012810 2429 mini_ranger.cc:160] Time spent starting Ranger: real 9.029s user 0.004s sys 0.039s
Sep 27, 2026 6:40:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:40:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 6:40:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:40:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:40:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:40:20 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 18:40:20.930495 2429 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.95.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.95.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/rangerkms_127.2.95.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/rangerkms_127.2.95.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.95.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.95.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.95.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/HTTP_127.2.95.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.95.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/HTTP_127.2.95.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 18:40:21 dist-test-slave-9jll krb5kdc[13323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534421, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 18:40:21.090890 2429 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 18:40:21.110904 2429 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 18:40:22.116991 2429 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 18:40:22.117147 2429 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-kms
I20260927 18:40:22.117251 2429 mini_ranger_kms.cc:227] Using postgres at 127.2.95.84:49719
2026-09-27 18:40:22,523 [I] DB FLAVOR :POSTGRES
2026-09-27 18:40:22,523 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:40:22,523 [I] Checking connection
2026-09-27 18:40:22,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 18:40:22,762 [I] connection success
2026-09-27 18:40:22,762 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 18:40:22,762 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 18:40:22,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 18:40:23,017 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 18:40:23,018 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:40:23,018 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 18:40:23,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:49719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 18:40:23,306 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 18:40:23.741849 2429 mini_ranger_kms.cc:310] Created kms service
I20260927 18:40:23.926553 2429 mini_ranger_kms.cc:326] Created kudu user
I20260927 18:40:23.975747 2429 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 18:40:23 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534421, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:40:24.408699 2429 mini_ranger_kms.cc:384] Added ranger policy
I20260927 18:40:24.409013 2429 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 18:40:24.409051 2429 mini_ranger_kms.cc:240] Using host: 127.2.95.84
I20260927 18:40:24.430493 2429 mini_ranger_kms.cc:276] Ranger KMS PID: 13716
I20260927 18:40:24.430675 2429 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.95.84:51107
Sep 27, 2026 6:40:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:40:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:40:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:40:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:40:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:40:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 18:40:26.504922 2429 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.414s user 0.123s sys 0.307s
I20260927 18:40:26.505178 2429 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 18:40:26.505412 2429 mini_ranger_kms.cc:401] 127.2.95.84:51107/kms/v1/keys
Sep 27 18:40:26 dist-test-slave-9jll krb5kdc[13323](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534426, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.95.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 18:40:27 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534426, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.95.84@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
Sep 27 18:40:27 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534426, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.95.84@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
Sep 27 18:40:29 dist-test-slave-9jll krb5kdc[13323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:40:29.902457 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:42173
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:42173
--ranger_config_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.95.84:51107/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:40:30.253306 13767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:40:30.253620 13767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:40:30.253769 13767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:40:30.261502 13767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:40:30.261629 13767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:40:30.261737 13767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:40:30.261811 13767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:40:30.261883 13767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:40:30.272671 13767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.95.84:51107/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:42173
--ranger_config_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:42173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:40:30.274231 13767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:40:30.275520 13767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:40:30.287832 13773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:40:30.287925 13775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:40:30.288579 13767 server_base.cc:1034] running on GCE node
W20260927 18:40:30.288687 13772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:40:30.289912 13767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:40:30.291458 13767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:40:30.292673 13767 hybrid_clock.cc:648] HybridClock initialized: now 1790534430292628 us; error 40 us; skew 500 ppm
I20260927 18:40:30.293018 13767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:40:30 dist-test-slave-9jll krb5kdc[13323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:40:30.302639 13767 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:40:30.304188 13767 webserver.cc:460] Webserver started at http://127.2.95.126:33701/ using document root <none> and password file <none>
I20260927 18:40:30.304858 13767 fs_manager.cc:359] Metadata directory not provided
I20260927 18:40:30.304973 13767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:40:30.305266 13767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:40:30 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534430, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:40:30.367975 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "0b3fa2fe613b44c7b9179532b5f4b340"
format_stamp: "Formatted at 2026-09-27 18:40:30 on dist-test-slave-9jll"
server_key: "20c8ebf2f56e6db7734b359678e852ec"
server_key_iv: "9131f00ed748346c568b41cbe79a9c4a"
server_key_version: "kuduclusterkey@0"
I20260927 18:40:30.368847 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "0b3fa2fe613b44c7b9179532b5f4b340"
format_stamp: "Formatted at 2026-09-27 18:40:30 on dist-test-slave-9jll"
server_key: "20c8ebf2f56e6db7734b359678e852ec"
server_key_iv: "9131f00ed748346c568b41cbe79a9c4a"
server_key_version: "kuduclusterkey@0"
I20260927 18:40:30.375087 13767 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 18:40:30.391522 13782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:40:30.393019 13767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 18:40:30.393213 13767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "0b3fa2fe613b44c7b9179532b5f4b340"
format_stamp: "Formatted at 2026-09-27 18:40:30 on dist-test-slave-9jll"
server_key: "20c8ebf2f56e6db7734b359678e852ec"
server_key_iv: "9131f00ed748346c568b41cbe79a9c4a"
server_key_version: "kuduclusterkey@0"
I20260927 18:40:30.393435 13767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:40:30.419175 13767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:40:30.425266 13767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:40:30.425616 13767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:40:30.445803 13767 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:42173
I20260927 18:40:30.445825 13833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:42173 every 8 connection(s)
I20260927 18:40:30.447186 13767 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:40:30.451123 13834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:40:30.452152 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 13767
I20260927 18:40:30.452368 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:40:30.453233 2429 external_mini_cluster.cc:1428] Setting key 0ae2c1d8df44479d59611fbc52c278c6
I20260927 18:40:30.464815 13834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340: Bootstrap starting.
Sep 27 18:40:30 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:40:30.468526 13834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340: Neither blocks nor log segments found. Creating new log.
I20260927 18:40:30.469727 13834 log.cc:824] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340: Log is configured to *not* fsync() on all Append() calls
I20260927 18:40:30.472891 13834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340: No bootstrap required, opened a new log
I20260927 18:40:30.475915 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:30.455782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.2.95.126:42173)
0927 18:40:30.456585 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:30.456599 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:30.456641 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:40:30.458398 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:30.458427 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:30.458902 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:30.459277 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:40:30.460627 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:30.462025 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:30.463245 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:30.463928 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:30.464088 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:30.468327 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:40:30.468336 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:40:30.468351 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:40:30.468443 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:40:30.471743 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:30.472484 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:30.472491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:30.472497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:30.472660 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:30.473070 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:30.473074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:30.473076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:30.473652 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:40:30.473966 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:30.474445 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 18:40:30.474887 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":338,"threads_started":1}
I20260927 18:40:30.481498 13834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3fa2fe613b44c7b9179532b5f4b340" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42173 } }
I20260927 18:40:30.482400 13834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:40:30.482501 13834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b3fa2fe613b44c7b9179532b5f4b340, State: Initialized, Role: FOLLOWER
I20260927 18:40:30.483083 13834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3fa2fe613b44c7b9179532b5f4b340" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42173 } }
I20260927 18:40:30.483287 13834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:40:30.483405 13834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:40:30.483560 13834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:40:30.485760 13834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3fa2fe613b44c7b9179532b5f4b340" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42173 } }
I20260927 18:40:30.486300 13834 leader_election.cc:302] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b3fa2fe613b44c7b9179532b5f4b340; no voters:
I20260927 18:40:30.486855 13834 leader_election.cc:288] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:40:30.486981 13839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:40:30.488026 13839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [term 1 LEADER]: Becoming Leader. State: Replica: 0b3fa2fe613b44c7b9179532b5f4b340, State: Running, Role: LEADER
I20260927 18:40:30.488686 13839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3fa2fe613b44c7b9179532b5f4b340" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42173 } }
I20260927 18:40:30.488848 13834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:40:30.491977 13841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b3fa2fe613b44c7b9179532b5f4b340. Latest consensus state: current_term: 1 leader_uuid: "0b3fa2fe613b44c7b9179532b5f4b340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3fa2fe613b44c7b9179532b5f4b340" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42173 } } }
I20260927 18:40:30.492529 13841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [sys.catalog]: This master's current role is: LEADER
I20260927 18:40:30.492205 13840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b3fa2fe613b44c7b9179532b5f4b340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b3fa2fe613b44c7b9179532b5f4b340" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 42173 } } }
I20260927 18:40:30.492985 13840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340 [sys.catalog]: This master's current role is: LEADER
I20260927 18:40:30.496881 13848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:40:30.497007 13834 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:40:30.511464 13848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:40:30.527398 13848 catalog_manager.cc:1347] Generated new cluster ID: 89d4a93ee0554292a1a355bf3c335099
I20260927 18:40:30.527550 13848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:40:30.544190 13848 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:40:30.546238 13848 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:40:30.560925 13848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340: Generated new TSK 0
I20260927 18:40:30.562165 13848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:40:31.258380 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.95.84:51107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42173
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:40:31 dist-test-slave-9jll krb5kdc[13323](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:40:31.683668 13879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:40:31.683957 13879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:40:31.684070 13879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:40:31.692171 13879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:40:31.692315 13879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:40:31.692497 13879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:40:31.702824 13879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.95.84:51107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:40:31.704149 13879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:40:31.705470 13879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:40:31.720643 13890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:40:31.720783 13889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:40:31.720760 13892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:40:31.722671 13879 server_base.cc:1034] running on GCE node
I20260927 18:40:31.723510 13879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:40:31.724812 13879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:40:31.726079 13879 hybrid_clock.cc:648] HybridClock initialized: now 1790534431726012 us; error 56 us; skew 500 ppm
I20260927 18:40:31.726488 13879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:40:31 dist-test-slave-9jll krb5kdc[13323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:40:31.730806 13879 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:40:31.733239 13879 webserver.cc:460] Webserver started at http://127.2.95.65:41409/ using document root <none> and password file <none>
I20260927 18:40:31.734254 13879 fs_manager.cc:359] Metadata directory not provided
I20260927 18:40:31.734403 13879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:40:31.734782 13879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:40:31 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:40:31.760387 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "1346335cb8d94d04baa5eef6df33001f"
format_stamp: "Formatted at 2026-09-27 18:40:31 on dist-test-slave-9jll"
server_key: "2993b53dcebdc148a3ca08877dbc2e29"
server_key_iv: "9beb4654ce11bee759b26671132131f3"
server_key_version: "kuduclusterkey@0"
I20260927 18:40:31.761847 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "1346335cb8d94d04baa5eef6df33001f"
format_stamp: "Formatted at 2026-09-27 18:40:31 on dist-test-slave-9jll"
server_key: "2993b53dcebdc148a3ca08877dbc2e29"
server_key_iv: "9beb4654ce11bee759b26671132131f3"
server_key_version: "kuduclusterkey@0"
I20260927 18:40:31.770304 13879 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.001s
I20260927 18:40:31.810201 13899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:40:31.812389 13879 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.003s
I20260927 18:40:31.812616 13879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "1346335cb8d94d04baa5eef6df33001f"
format_stamp: "Formatted at 2026-09-27 18:40:31 on dist-test-slave-9jll"
server_key: "2993b53dcebdc148a3ca08877dbc2e29"
server_key_iv: "9beb4654ce11bee759b26671132131f3"
server_key_version: "kuduclusterkey@0"
I20260927 18:40:31.812912 13879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:40:31.837353 13879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:40:31.842717 13879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:40:31.843135 13879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:40:31.844446 13879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:40:31.846501 13879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:40:31.846603 13879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:40:31.846781 13879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:40:31.846848 13879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:40:31.912655 13879 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:37291
I20260927 18:40:31.912665 14011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:37291 every 8 connection(s)
I20260927 18:40:31.914392 13879 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:40:31.922219 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 13879
I20260927 18:40:31.922466 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:40:31.923262 2429 external_mini_cluster.cc:1428] Setting key 03b99f17e497eb6289e022ad57960403
Sep 27 18:40:31 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
I20260927 18:40:31.958510 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:31.919006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:55333 (local address 127.2.95.126:42173)
0927 18:40:31.933488 (+ 14482us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:31.933496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:31.937690 (+ 4194us) server_negotiation.cc:407] Connection header received
0927 18:40:31.937766 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:31.937772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:31.937887 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:31.937985 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:40:31.940282 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:31.941297 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:31.943757 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:31.944345 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:31.944442 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:31.949272 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:40:31.949291 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:40:31.949297 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:40:31.949345 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:40:31.953714 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:31.954902 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:31.954909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:31.954913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:31.955086 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:31.955689 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:31.955694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:31.955698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:31.956066 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:40:31.956213 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:31.957890 (+ 1677us) server_negotiation.cc:299] Negotiation successful
0927 18:40:31.958131 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14304,"thread_start_us":14483,"threads_started":1}
I20260927 18:40:31.959210 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:31.918185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:42173 (local address 127.2.95.65:55333)
0927 18:40:31.934807 (+ 16622us) negotiation.cc:107] Waiting for socket to connect
0927 18:40:31.934888 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 18:40:31.937239 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:40:31.938458 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:40:31.938472 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:40:31.939112 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:40:31.940016 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:31.940048 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:31.941538 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:31.941543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:31.943559 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:31.943577 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:31.944824 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:31.944831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:31.945129 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:31.945983 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:40:31.946013 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:40:31.948883 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:40:31.953966 (+ 5083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:40:31.953979 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:40:31.954001 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:40:31.954701 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:40:31.955240 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:40:31.955246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:40:31.955250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:40:31.955544 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:40:31.956292 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:40:31.956302 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:40:31.956833 (+ 531us) client_negotiation.cc:769] Sending connection context
0927 18:40:31.957150 (+ 317us) client_negotiation.cc:240] Negotiation successful
0927 18:40:31.957785 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16293,"thread_start_us":16024,"threads_started":1}
I20260927 18:40:31.961373 14012 heartbeater.cc:347] Connected to a master server at 127.2.95.126:42173
I20260927 18:40:31.961745 14012 heartbeater.cc:464] Registering TS with master...
I20260927 18:40:31.962666 14012 heartbeater.cc:511] Master 127.2.95.126:42173 requested a full tablet report, sending...
I20260927 18:40:31.965543 13799 ts_manager.cc:194] Registered new tserver with Master: 1346335cb8d94d04baa5eef6df33001f (127.2.95.65:37291)
I20260927 18:40:31.967676 13799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:55333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:40:31.991170 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.95.84:51107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42173
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:40:32 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
Sep 27 18:40:32 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
W20260927 18:40:32.497774 14018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:40:32.498302 14018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:40:32.498500 14018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:40:32.509378 14018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:40:32.509622 14018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:40:32.509884 14018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:40:32.526021 14018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.95.84:51107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:40:32.528096 14018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:40:32.529956 14018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:40:32.576453 14028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:40:32.577039 14025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:40:32.582182 14026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:40:32.587550 14018 server_base.cc:1034] running on GCE node
I20260927 18:40:32.588747 14018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:40:32.599550 14018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:40:32.601193 14018 hybrid_clock.cc:648] HybridClock initialized: now 1790534432601069 us; error 132 us; skew 500 ppm
I20260927 18:40:32.601739 14018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:40:32 dist-test-slave-9jll krb5kdc[13323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:40:32.606313 14018 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:40:32.627683 14018 webserver.cc:460] Webserver started at http://127.2.95.66:33107/ using document root <none> and password file <none>
I20260927 18:40:32.628856 14018 fs_manager.cc:359] Metadata directory not provided
I20260927 18:40:32.629107 14018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:40:32.629557 14018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:40:32 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:40:32.657121 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "05d1f7efbc5f437595714c6722709f5f"
format_stamp: "Formatted at 2026-09-27 18:40:32 on dist-test-slave-9jll"
server_key: "df9ff93107ca89de289093d0c13e7cbd"
server_key_iv: "31006ccf82059bf5a3aa473656098da6"
server_key_version: "kuduclusterkey@0"
I20260927 18:40:32.658581 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "05d1f7efbc5f437595714c6722709f5f"
format_stamp: "Formatted at 2026-09-27 18:40:32 on dist-test-slave-9jll"
server_key: "df9ff93107ca89de289093d0c13e7cbd"
server_key_iv: "31006ccf82059bf5a3aa473656098da6"
server_key_version: "kuduclusterkey@0"
I20260927 18:40:32.667037 14018 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.001s
I20260927 18:40:32.700388 14035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:40:32.702886 14018 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.007s
I20260927 18:40:32.703262 14018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "05d1f7efbc5f437595714c6722709f5f"
format_stamp: "Formatted at 2026-09-27 18:40:32 on dist-test-slave-9jll"
server_key: "df9ff93107ca89de289093d0c13e7cbd"
server_key_iv: "31006ccf82059bf5a3aa473656098da6"
server_key_version: "kuduclusterkey@0"
I20260927 18:40:32.703648 14018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:40:32.743427 14018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:40:32.749590 14018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:40:32.750113 14018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:40:32.758600 14018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:40:32.764773 14018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:40:32.765285 14018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260927 18:40:32.765496 14018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:40:32.765599 14018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:40:32.826068 14018 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:36803
I20260927 18:40:32.826112 14154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:36803 every 8 connection(s)
I20260927 18:40:32.827904 14018 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:40:32.835192 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 14018
I20260927 18:40:32.835394 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:40:32.835979 2429 external_mini_cluster.cc:1428] Setting key f5b5d31b2de0a3f402bab9faeb145697
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
Sep 27 18:40:32 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:40:32.877167 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:32.830960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:54643 (local address 127.2.95.126:42173)
0927 18:40:32.846357 (+ 15397us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:32.846364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:32.849347 (+ 2983us) server_negotiation.cc:407] Connection header received
0927 18:40:32.849431 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:32.849436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:32.849541 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:32.849663 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:40:32.853833 (+ 4170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:32.854951 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:32.857525 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:32.858213 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:32.858355 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:32.869646 (+ 11291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:40:32.869654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:40:32.869660 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:40:32.869737 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:40:32.873102 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:32.874270 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:32.874277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:32.874281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:32.874476 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:32.875053 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:32.875058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:32.875062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:32.875446 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:40:32.875595 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:32.876510 (+ 915us) server_negotiation.cc:299] Negotiation successful
0927 18:40:32.876765 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15247,"thread_start_us":16350,"threads_started":1}
I20260927 18:40:32.878378 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:32.831594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:42173 (local address 127.2.95.66:54643)
0927 18:40:32.846960 (+ 15366us) negotiation.cc:107] Waiting for socket to connect
0927 18:40:32.847024 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 18:40:32.849067 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:40:32.849996 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:40:32.850009 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:40:32.850760 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:40:32.851617 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:32.851643 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:32.855177 (+ 3534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:32.855184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:32.857302 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:32.857322 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:32.858489 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:32.858497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:32.858794 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:32.866160 (+ 7366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:40:32.866316 (+ 156us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:40:32.869282 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:40:32.873327 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:40:32.873339 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:40:32.873361 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:40:32.874028 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:40:32.874637 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:40:32.874642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:40:32.874646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:40:32.874931 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:40:32.875592 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:40:32.875600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:40:32.876063 (+ 463us) client_negotiation.cc:769] Sending connection context
0927 18:40:32.876379 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 18:40:32.876915 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15003,"thread_start_us":14788,"threads_started":1}
I20260927 18:40:32.880035 14155 heartbeater.cc:347] Connected to a master server at 127.2.95.126:42173
I20260927 18:40:32.880401 14155 heartbeater.cc:464] Registering TS with master...
I20260927 18:40:32.881274 14155 heartbeater.cc:511] Master 127.2.95.126:42173 requested a full tablet report, sending...
I20260927 18:40:32.883136 13799 ts_manager.cc:194] Registered new tserver with Master: 05d1f7efbc5f437595714c6722709f5f (127.2.95.66:36803)
I20260927 18:40:32.884341 13799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:54643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:40:32.897771 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.95.84:51107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42173
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:40:32.972118 14012 heartbeater.cc:503] Master 127.2.95.126:42173 was elected leader, sending a full tablet report...
W20260927 18:40:33.262295 14161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:40:33.262593 14161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:40:33.262687 14161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:40:33.270018 14161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:40:33.270133 14161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:40:33.270268 14161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:40:33.280622 14161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.95.84:51107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:42173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:40:33.281952 14161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:40:33.283622 14161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:40:33.297365 14166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:40:33.297425 14167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:40:33.297400 14169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:40:33.298431 14161 server_base.cc:1034] running on GCE node
I20260927 18:40:33.299100 14161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:40:33.300099 14161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:40:33.301367 14161 hybrid_clock.cc:648] HybridClock initialized: now 1790534433301262 us; error 88 us; skew 500 ppm
I20260927 18:40:33.301783 14161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:40:33 dist-test-slave-9jll krb5kdc[13323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534433, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:40:33.305167 14161 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:40:33.306906 14161 webserver.cc:460] Webserver started at http://127.2.95.67:43697/ using document root <none> and password file <none>
I20260927 18:40:33.307596 14161 fs_manager.cc:359] Metadata directory not provided
I20260927 18:40:33.307715 14161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:40:33.308008 14161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:40:33 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534433, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:40:33.325652 14161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "3db1ce7f3e0e4b9e822069b40c019de3"
format_stamp: "Formatted at 2026-09-27 18:40:33 on dist-test-slave-9jll"
server_key: "97065ed97763076a13684c09af2008b7"
server_key_iv: "e67f04bebd66587afa6f595997ddeea8"
server_key_version: "kuduclusterkey@0"
I20260927 18:40:33.326646 14161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "3db1ce7f3e0e4b9e822069b40c019de3"
format_stamp: "Formatted at 2026-09-27 18:40:33 on dist-test-slave-9jll"
server_key: "97065ed97763076a13684c09af2008b7"
server_key_iv: "e67f04bebd66587afa6f595997ddeea8"
server_key_version: "kuduclusterkey@0"
I20260927 18:40:33.332688 14161 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:40:33.347699 14176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:40:33.349380 14161 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 18:40:33.349571 14161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "3db1ce7f3e0e4b9e822069b40c019de3"
format_stamp: "Formatted at 2026-09-27 18:40:33 on dist-test-slave-9jll"
server_key: "97065ed97763076a13684c09af2008b7"
server_key_iv: "e67f04bebd66587afa6f595997ddeea8"
server_key_version: "kuduclusterkey@0"
I20260927 18:40:33.349833 14161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:40:33.374393 14161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:40:33.379184 14161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:40:33.379531 14161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:40:33.380633 14161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:40:33.382167 14161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:40:33.382246 14161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:40:33.382352 14161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:40:33.382401 14161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:40:33.414564 14161 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:39271
I20260927 18:40:33.414608 14288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:39271 every 8 connection(s)
I20260927 18:40:33.415813 14161 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:40:33.422807 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 14161
I20260927 18:40:33.422986 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:40:33.423412 2429 external_mini_cluster.cc:1428] Setting key bd2c74f35d492d4039426623850a229d
Sep 27 18:40:33 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534433, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:40:33.437279 14291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.418514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:33251 (local address 127.2.95.126:42173)
0927 18:40:33.418988 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:33.418993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:33.419937 (+ 944us) server_negotiation.cc:407] Connection header received
0927 18:40:33.422380 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:33.422387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:33.422500 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:33.422614 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:40:33.424292 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.425050 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.426702 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.427130 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.427265 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.430856 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:40:33.430861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:40:33.430864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:40:33.430901 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:40:33.433931 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:33.434770 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:33.434774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:33.434776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:33.434906 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:33.435420 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:33.435426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:33.435428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:33.435765 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:40:33.435921 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:33.436665 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 18:40:33.436902 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":224,"threads_started":1}
I20260927 18:40:33.438906 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.418930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:42173 (local address 127.2.95.67:33251)
0927 18:40:33.419712 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0927 18:40:33.419792 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 18:40:33.422023 (+ 2231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:40:33.422934 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:40:33.422948 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:40:33.423437 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:40:33.424094 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:33.424119 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.425212 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.425218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:33.426540 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:33.426554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.427368 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.427375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:33.427611 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.428399 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:40:33.428425 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:40:33.430564 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:40:33.434112 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:40:33.434124 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:40:33.434141 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:40:33.434624 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:40:33.435020 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:40:33.435025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:40:33.435029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:40:33.435271 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:40:33.435903 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:40:33.435912 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:40:33.436272 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 18:40:33.436548 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 18:40:33.437111 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":329,"threads_started":1}
I20260927 18:40:33.440912 14289 heartbeater.cc:347] Connected to a master server at 127.2.95.126:42173
I20260927 18:40:33.441259 14289 heartbeater.cc:464] Registering TS with master...
I20260927 18:40:33.442080 14289 heartbeater.cc:511] Master 127.2.95.126:42173 requested a full tablet report, sending...
I20260927 18:40:33.443643 13799 ts_manager.cc:194] Registered new tserver with Master: 3db1ce7f3e0e4b9e822069b40c019de3 (127.2.95.67:39271)
I20260927 18:40:33.444559 13799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:33251
I20260927 18:40:33.452013 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:40:33.470531 14291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.456971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.2.95.126:42173)
0927 18:40:33.457142 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:33.457146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:33.457770 (+ 624us) server_negotiation.cc:407] Connection header received
0927 18:40:33.457977 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:33.457981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:33.458062 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:33.458148 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:40:33.459672 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.460442 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.461875 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.462325 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.462424 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.464896 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:40:33.464901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:40:33.464904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:40:33.464943 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:40:33.467678 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:33.468484 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:33.468490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:33.468494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:33.468633 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:33.469035 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:33.469040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:33.469043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:33.469336 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:40:33.469574 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:33.469949 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 18:40:33.470196 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:40:33.474983 13799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:40:33.482496 13799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:40:33.508505 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.495744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:37291 (local address 127.0.0.1:36308)
0927 18:40:33.498114 (+ 2370us) negotiation.cc:107] Waiting for socket to connect
0927 18:40:33.498171 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 18:40:33.498390 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:40:33.499596 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:40:33.499605 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:40:33.499655 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:40:33.500092 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:33.500108 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.502387 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.502394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:33.505080 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:33.505103 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.507509 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.507514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:33.507664 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.507699 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:40:33.507830 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 18:40:33.508067 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2065,"thread_start_us":10649,"threads_started":1}
I20260927 18:40:33.508723 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.496041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.2.95.65:37291)
0927 18:40:33.497022 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:33.497032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:33.498608 (+ 1576us) server_negotiation.cc:407] Connection header received
0927 18:40:33.498732 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:33.498741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:33.499054 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:33.499252 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:40:33.500333 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.502180 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.505324 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.506995 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.507125 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.507383 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:33.508177 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 18:40:33.508385 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":658,"threads_started":1}
I20260927 18:40:33.513114 13947 tablet_service.cc:1467] Processing CreateTablet for tablet f10db939aec2478bb904055b88c114c6 (DEFAULT_TABLE table=test-table [id=602acbe5dd1b4021984e8123c5c52d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:40:33.514745 13947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10db939aec2478bb904055b88c114c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:40:33.515486 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.495937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:36803 (local address 127.0.0.1:45480)
0927 18:40:33.506705 (+ 10768us) negotiation.cc:107] Waiting for socket to connect
0927 18:40:33.506728 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:40:33.506903 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:40:33.507853 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:40:33.507857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:40:33.507884 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:40:33.508266 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:33.508278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.510189 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.510195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:33.512831 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:33.512844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.514519 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.514526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:33.514693 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.514728 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:40:33.514815 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 18:40:33.515064 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10623,"mutex_wait_us":32,"thread_start_us":10871,"threads_started":1}
I20260927 18:40:33.516179 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.496436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:39271 (local address 127.0.0.1:44656)
0927 18:40:33.507044 (+ 10608us) negotiation.cc:107] Waiting for socket to connect
0927 18:40:33.507060 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:40:33.507254 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:40:33.508202 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:40:33.508209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:40:33.508238 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:40:33.508688 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:33.508701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.511074 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.511080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:33.513288 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:33.513303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.515379 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.515386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:33.515570 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.515604 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:40:33.515696 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 18:40:33.515828 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10494,"mutex_wait_us":1,"thread_start_us":1369,"threads_started":1}
I20260927 18:40:33.516727 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.496807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.2.95.66:36803)
0927 18:40:33.505325 (+ 8518us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:33.505335 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:33.507093 (+ 1758us) server_negotiation.cc:407] Connection header received
0927 18:40:33.507223 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:33.507231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:33.507650 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:33.507868 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:40:33.508505 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.510000 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.512987 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.514306 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.514442 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.516171 (+ 1729us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:33.516270 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 18:40:33.516447 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8340,"thread_start_us":8208,"threads_started":1}
I20260927 18:40:33.517733 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.496798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.2.95.67:39271)
0927 18:40:33.506219 (+ 9421us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:33.506229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:33.507147 (+ 918us) server_negotiation.cc:407] Connection header received
0927 18:40:33.507452 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:33.507459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:33.507647 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:33.507849 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:40:33.508929 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.510884 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.513459 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.514872 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.514987 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.515220 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:33.516443 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0927 18:40:33.516601 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9244,"thread_start_us":9077,"threads_started":1}
I20260927 18:40:33.520876 14224 tablet_service.cc:1467] Processing CreateTablet for tablet f10db939aec2478bb904055b88c114c6 (DEFAULT_TABLE table=test-table [id=602acbe5dd1b4021984e8123c5c52d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:40:33.520566 14090 tablet_service.cc:1467] Processing CreateTablet for tablet f10db939aec2478bb904055b88c114c6 (DEFAULT_TABLE table=test-table [id=602acbe5dd1b4021984e8123c5c52d13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:40:33.522545 14224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10db939aec2478bb904055b88c114c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:40:33.522558 14090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10db939aec2478bb904055b88c114c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:40:33.532033 14308 tablet_bootstrap.cc:492] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f: Bootstrap starting.
I20260927 18:40:33.535491 14309 tablet_bootstrap.cc:492] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f: Bootstrap starting.
I20260927 18:40:33.536062 14308 tablet_bootstrap.cc:654] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f: Neither blocks nor log segments found. Creating new log.
I20260927 18:40:33.537037 14310 tablet_bootstrap.cc:492] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3: Bootstrap starting.
I20260927 18:40:33.537499 14308 log.cc:824] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:40:33.539913 14309 tablet_bootstrap.cc:654] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f: Neither blocks nor log segments found. Creating new log.
I20260927 18:40:33.540566 14310 tablet_bootstrap.cc:654] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3: Neither blocks nor log segments found. Creating new log.
I20260927 18:40:33.541261 14309 log.cc:824] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:40:33.541738 14310 log.cc:824] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:40:33.544456 14308 tablet_bootstrap.cc:492] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f: No bootstrap required, opened a new log
I20260927 18:40:33.544617 14309 tablet_bootstrap.cc:492] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f: No bootstrap required, opened a new log
I20260927 18:40:33.544616 14310 tablet_bootstrap.cc:492] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3: No bootstrap required, opened a new log
I20260927 18:40:33.544761 14308 ts_tablet_manager.cc:1397] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260927 18:40:33.544895 14309 ts_tablet_manager.cc:1397] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 18:40:33.544885 14310 ts_tablet_manager.cc:1397] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 18:40:33.552735 14308 raft_consensus.cc:348] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1346335cb8d94d04baa5eef6df33001f" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 37291 } } peers { permanent_uuid: "05d1f7efbc5f437595714c6722709f5f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36803 } } peers { permanent_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 39271 } }
I20260927 18:40:33.553239 14308 raft_consensus.cc:374] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:40:33.553375 14308 raft_consensus.cc:729] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1346335cb8d94d04baa5eef6df33001f, State: Initialized, Role: FOLLOWER
I20260927 18:40:33.554105 14308 consensus_queue.cc:260] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1346335cb8d94d04baa5eef6df33001f" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 37291 } } peers { permanent_uuid: "05d1f7efbc5f437595714c6722709f5f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36803 } } peers { permanent_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 39271 } }
I20260927 18:40:33.556263 14308 ts_tablet_manager.cc:1428] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260927 18:40:33.556120 14309 raft_consensus.cc:348] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1346335cb8d94d04baa5eef6df33001f" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 37291 } } peers { permanent_uuid: "05d1f7efbc5f437595714c6722709f5f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36803 } } peers { permanent_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 39271 } }
I20260927 18:40:33.556062 14310 raft_consensus.cc:348] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1346335cb8d94d04baa5eef6df33001f" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 37291 } } peers { permanent_uuid: "05d1f7efbc5f437595714c6722709f5f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36803 } } peers { permanent_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 39271 } }
I20260927 18:40:33.556532 14310 raft_consensus.cc:374] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:40:33.556541 14309 raft_consensus.cc:374] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:40:33.556638 14310 raft_consensus.cc:729] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db1ce7f3e0e4b9e822069b40c019de3, State: Initialized, Role: FOLLOWER
I20260927 18:40:33.556638 14309 raft_consensus.cc:729] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d1f7efbc5f437595714c6722709f5f, State: Initialized, Role: FOLLOWER
I20260927 18:40:33.557273 14309 consensus_queue.cc:260] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1346335cb8d94d04baa5eef6df33001f" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 37291 } } peers { permanent_uuid: "05d1f7efbc5f437595714c6722709f5f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36803 } } peers { permanent_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 39271 } }
I20260927 18:40:33.557294 14310 consensus_queue.cc:260] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1346335cb8d94d04baa5eef6df33001f" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 37291 } } peers { permanent_uuid: "05d1f7efbc5f437595714c6722709f5f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36803 } } peers { permanent_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 39271 } }
I20260927 18:40:33.559122 14155 heartbeater.cc:503] Master 127.2.95.126:42173 was elected leader, sending a full tablet report...
I20260927 18:40:33.560705 14309 ts_tablet_manager.cc:1428] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260927 18:40:33.562960 14289 heartbeater.cc:503] Master 127.2.95.126:42173 was elected leader, sending a full tablet report...
I20260927 18:40:33.564178 14310 ts_tablet_manager.cc:1428] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3: Time spent starting tablet: real 0.019s user 0.012s sys 0.007s
W20260927 18:40:33.588305 14156 tablet.cc:2369] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:40:33.667426 14290 tablet.cc:2369] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:40:33.673416 14013 tablet.cc:2369] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:40:33.719158 14316 raft_consensus.cc:482] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:40:33.719998 14316 raft_consensus.cc:504] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1346335cb8d94d04baa5eef6df33001f" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 37291 } } peers { permanent_uuid: "05d1f7efbc5f437595714c6722709f5f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36803 } } peers { permanent_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 39271 } }
I20260927 18:40:33.721946 14316 leader_election.cc:288] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1346335cb8d94d04baa5eef6df33001f (127.2.95.65:37291), 05d1f7efbc5f437595714c6722709f5f (127.2.95.66:36803)
I20260927 18:40:33.729338 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.722338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:48961 (local address 127.2.95.65:37291)
0927 18:40:33.722592 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:33.722612 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:33.722660 (+ 48us) server_negotiation.cc:407] Connection header received
0927 18:40:33.722739 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:33.722744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:33.722827 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:33.722930 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:40:33.723853 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.724881 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.727196 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.728410 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.728563 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.728657 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:33.729025 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 18:40:33.729152 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:40:33.729447 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.722260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:37291 (local address 127.2.95.67:48961)
0927 18:40:33.722427 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 18:40:33.722448 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:40:33.722588 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:40:33.723166 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:40:33.723174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:40:33.723238 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:40:33.723613 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:33.723644 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.725123 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.725129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:33.726998 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:33.727010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.728629 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.728635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:33.728829 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.728866 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:40:33.728973 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:40:33.729126 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":1}
I20260927 18:40:33.730767 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.722261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:36803 (local address 127.2.95.67:35439)
0927 18:40:33.723635 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0927 18:40:33.723653 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:40:33.723781 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:40:33.724270 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:40:33.724275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:40:33.724305 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:40:33.724684 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:33.724696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.726437 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.726442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:33.728397 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:40:33.728412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.730089 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.730095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:40:33.730274 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.730299 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:40:33.730382 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 18:40:33.730494 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1260,"mutex_wait_us":429,"thread_start_us":624,"threads_started":1}
I20260927 18:40:33.730851 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.722646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:35439 (local address 127.2.95.66:36803)
0927 18:40:33.723028 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:33.723034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:33.723769 (+ 735us) server_negotiation.cc:407] Connection header received
0927 18:40:33.723993 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:33.723998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:33.724125 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:33.724210 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:40:33.724902 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.726212 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.728631 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.729891 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.730006 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.730079 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:33.730432 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 18:40:33.730567 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260927 18:40:33.730862 13967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10db939aec2478bb904055b88c114c6" candidate_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1346335cb8d94d04baa5eef6df33001f" is_pre_election: true
I20260927 18:40:33.731523 13967 raft_consensus.cc:2454] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db1ce7f3e0e4b9e822069b40c019de3 in term 0.
I20260927 18:40:33.732484 14179 leader_election.cc:302] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1346335cb8d94d04baa5eef6df33001f, 3db1ce7f3e0e4b9e822069b40c019de3; no voters:
I20260927 18:40:33.731889 14110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10db939aec2478bb904055b88c114c6" candidate_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d1f7efbc5f437595714c6722709f5f" is_pre_election: true
I20260927 18:40:33.732925 14316 raft_consensus.cc:2781] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:40:33.732944 14110 raft_consensus.cc:2454] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db1ce7f3e0e4b9e822069b40c019de3 in term 0.
I20260927 18:40:33.733083 14316 raft_consensus.cc:482] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:40:33.733188 14316 raft_consensus.cc:3037] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:40:33.735419 14316 raft_consensus.cc:504] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1346335cb8d94d04baa5eef6df33001f" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 37291 } } peers { permanent_uuid: "05d1f7efbc5f437595714c6722709f5f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36803 } } peers { permanent_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 39271 } }
I20260927 18:40:33.736027 14316 leader_election.cc:288] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [CANDIDATE]: Term 1 election: Requested vote from peers 1346335cb8d94d04baa5eef6df33001f (127.2.95.65:37291), 05d1f7efbc5f437595714c6722709f5f (127.2.95.66:36803)
I20260927 18:40:33.736677 14110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10db939aec2478bb904055b88c114c6" candidate_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d1f7efbc5f437595714c6722709f5f"
I20260927 18:40:33.736677 13967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10db939aec2478bb904055b88c114c6" candidate_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1346335cb8d94d04baa5eef6df33001f"
I20260927 18:40:33.736905 13967 raft_consensus.cc:3037] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:40:33.736946 14110 raft_consensus.cc:3037] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:40:33.740183 14110 raft_consensus.cc:2454] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db1ce7f3e0e4b9e822069b40c019de3 in term 1.
I20260927 18:40:33.740183 13967 raft_consensus.cc:2454] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db1ce7f3e0e4b9e822069b40c019de3 in term 1.
I20260927 18:40:33.740734 14177 leader_election.cc:302] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05d1f7efbc5f437595714c6722709f5f, 1346335cb8d94d04baa5eef6df33001f, 3db1ce7f3e0e4b9e822069b40c019de3; no voters:
I20260927 18:40:33.741214 14316 raft_consensus.cc:2781] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:40:33.741732 14316 raft_consensus.cc:686] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 1 LEADER]: Becoming Leader. State: Replica: 3db1ce7f3e0e4b9e822069b40c019de3, State: Running, Role: LEADER
I20260927 18:40:33.742229 14316 consensus_queue.cc:237] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1346335cb8d94d04baa5eef6df33001f" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 37291 } } peers { permanent_uuid: "05d1f7efbc5f437595714c6722709f5f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36803 } } peers { permanent_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 39271 } }
I20260927 18:40:33.747502 13798 catalog_manager.cc:5495] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3db1ce7f3e0e4b9e822069b40c019de3 (127.2.95.67). New cstate: current_term: 1 leader_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1346335cb8d94d04baa5eef6df33001f" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05d1f7efbc5f437595714c6722709f5f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db1ce7f3e0e4b9e822069b40c019de3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 39271 } health_report { overall_health: HEALTHY } } }
I20260927 18:40:33.818403 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.812955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.2.95.67:39271)
0927 18:40:33.813127 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:33.813132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:33.813197 (+ 65us) server_negotiation.cc:407] Connection header received
0927 18:40:33.813363 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:33.813367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:33.813455 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:33.813579 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:40:33.813983 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.814704 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.816240 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.816633 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.816724 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.817276 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:40:33.817840 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:40:33.817979 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:33.818126 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 18:40:33.818196 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:40:33.827397 13967 raft_consensus.cc:1261] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f [term 1 FOLLOWER]: Refusing update from remote peer 3db1ce7f3e0e4b9e822069b40c019de3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:40:33.827404 14110 raft_consensus.cc:1261] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f [term 1 FOLLOWER]: Refusing update from remote peer 3db1ce7f3e0e4b9e822069b40c019de3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:40:33.828562 14318 consensus_queue.cc:1035] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1346335cb8d94d04baa5eef6df33001f" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:40:33.829540 14316 consensus_queue.cc:1035] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05d1f7efbc5f437595714c6722709f5f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:40:33.845784 14323 mvcc.cc:205] Tried to move back new op lower bound from 7334029040944562176 to 7334029040611602432. Current Snapshot: MvccSnapshot[applied={T|T < 7334029040944562176}]
I20260927 18:40:33.846481 14321 mvcc.cc:205] Tried to move back new op lower bound from 7334029040944562176 to 7334029040611602432. Current Snapshot: MvccSnapshot[applied={T|T < 7334029040944562176}]
I20260927 18:40:33.853147 14322 mvcc.cc:205] Tried to move back new op lower bound from 7334029040944562176 to 7334029040611602432. Current Snapshot: MvccSnapshot[applied={T|T < 7334029040944562176}]
I20260927 18:40:33.860194 13798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:40:33.860451 13798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:40:33.864451 13798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b3fa2fe613b44c7b9179532b5f4b340: Sending DeleteTablet for 3 replicas of tablet f10db939aec2478bb904055b88c114c6
I20260927 18:40:33.865734 13947 tablet_service.cc:1514] Processing DeleteTablet for tablet f10db939aec2478bb904055b88c114c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:40:33 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:36308
I20260927 18:40:33.865734 14224 tablet_service.cc:1514] Processing DeleteTablet for tablet f10db939aec2478bb904055b88c114c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:40:33 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:44656
I20260927 18:40:33.865762 14090 tablet_service.cc:1514] Processing DeleteTablet for tablet f10db939aec2478bb904055b88c114c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:40:33 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:45480
I20260927 18:40:33.872916 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 13879
I20260927 18:40:33.881179 14330 tablet_replica.cc:330] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3: stopping tablet replica
I20260927 18:40:33.881721 14330 raft_consensus.cc:2229] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:40:33.882215 14331 tablet_replica.cc:330] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f: stopping tablet replica
I20260927 18:40:33.882409 14332 tablet_replica.cc:330] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f: stopping tablet replica
I20260927 18:40:33.882699 14331 raft_consensus.cc:2229] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:40:33.882903 14332 raft_consensus.cc:2229] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:40:33.883294 14331 raft_consensus.cc:2258] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:40:33.883488 14332 raft_consensus.cc:2258] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:40:33.885608 14331 ts_tablet_manager.cc:1905] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:40:33.885607 14332 ts_tablet_manager.cc:1905] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:40:33.890197 14330 raft_consensus.cc:2258] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:40:33.892791 14330 ts_tablet_manager.cc:1905] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:40:33.893045 14332 ts_tablet_manager.cc:1918] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:40:33.893342 14332 log.cc:1196] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/f10db939aec2478bb904055b88c114c6
I20260927 18:40:33.893476 14331 ts_tablet_manager.cc:1918] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:40:33.893712 14331 log.cc:1196] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/f10db939aec2478bb904055b88c114c6
I20260927 18:40:33.894106 14332 ts_tablet_manager.cc:1939] T f10db939aec2478bb904055b88c114c6 P 05d1f7efbc5f437595714c6722709f5f: Deleting consensus metadata
I20260927 18:40:33.894294 14331 ts_tablet_manager.cc:1939] T f10db939aec2478bb904055b88c114c6 P 1346335cb8d94d04baa5eef6df33001f: Deleting consensus metadata
I20260927 18:40:33.896175 13785 catalog_manager.cc:4841] TS 1346335cb8d94d04baa5eef6df33001f (127.2.95.65:37291): tablet f10db939aec2478bb904055b88c114c6 (table test-table [id=602acbe5dd1b4021984e8123c5c52d13]) successfully deleted
I20260927 18:40:33.896575 13783 catalog_manager.cc:4841] TS 05d1f7efbc5f437595714c6722709f5f (127.2.95.66:36803): tablet f10db939aec2478bb904055b88c114c6 (table test-table [id=602acbe5dd1b4021984e8123c5c52d13]) successfully deleted
Sep 27 18:40:33 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:40:33.900058 14330 ts_tablet_manager.cc:1918] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:40:33.900260 14330 log.cc:1196] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/f10db939aec2478bb904055b88c114c6
I20260927 18:40:33.900807 14330 ts_tablet_manager.cc:1939] T f10db939aec2478bb904055b88c114c6 P 3db1ce7f3e0e4b9e822069b40c019de3: Deleting consensus metadata
I20260927 18:40:33.902773 13786 catalog_manager.cc:4841] TS 3db1ce7f3e0e4b9e822069b40c019de3 (127.2.95.67:39271): tablet f10db939aec2478bb904055b88c114c6 (table test-table [id=602acbe5dd1b4021984e8123c5c52d13]) successfully deleted
I20260927 18:40:33.907666 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:33.873956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.2.95.65:37291)
0927 18:40:33.874171 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:33.874177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:33.881510 (+ 7333us) server_negotiation.cc:407] Connection header received
0927 18:40:33.881729 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:33.881734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:33.881865 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:33.881983 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:40:33.884419 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.885475 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.887593 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:33.888325 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:33.888434 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:33.900100 (+ 11666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:40:33.900121 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:40:33.900142 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:40:33.900195 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:40:33.903828 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:33.904887 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:33.904895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:33.904897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:33.905492 (+ 595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:33.906027 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:33.906032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:33.906034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:33.906502 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:40:33.906735 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:33.907116 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 18:40:33.907297 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:40:33.908392 14007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:40:33.998207 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 13879
I20260927 18:40:34.019524 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 14018
Sep 27 18:40:34 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:40:34.035401 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:34.020490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.2.95.66:36803)
0927 18:40:34.020682 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:34.020688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:34.020708 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:40:34.020773 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:34.020777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:34.020861 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:34.020959 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:40:34.022398 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:34.023136 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:34.024424 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:34.024824 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:34.024932 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:34.029510 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:40:34.029527 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:40:34.029543 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:40:34.029581 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:40:34.032455 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:34.033227 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:34.033234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:34.033236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:34.033377 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:34.033818 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:34.033821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:34.033823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:34.034201 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:40:34.034490 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:34.034834 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 18:40:34.035051 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:40:34.036118 14150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:40:34.126039 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 14018
I20260927 18:40:34.147351 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 14161
Sep 27 18:40:34 dist-test-slave-9jll krb5kdc[13323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:40:34.163579 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:40:34.148106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.2.95.67:39271)
0927 18:40:34.148278 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 18:40:34.148285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:40:34.148376 (+ 91us) server_negotiation.cc:407] Connection header received
0927 18:40:34.148555 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:40:34.148559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:40:34.148621 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:40:34.148705 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:40:34.149993 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:34.150800 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:34.152136 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:40:34.152599 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:40:34.152732 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:40:34.157579 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:40:34.157594 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:40:34.157609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:40:34.157657 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:40:34.160658 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:34.161448 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:34.161455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:34.161458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:34.161609 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:40:34.162024 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:40:34.162028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:40:34.162031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:40:34.162408 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:40:34.162713 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 18:40:34.163027 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 18:40:34.163235 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:40:34.164361 14284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:40:34.285804 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 14161
I20260927 18:40:34.301448 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 13767
I20260927 18:40:34.302675 13829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:40:34.417151 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 13767
I20260927 18:40:34.438531 2429 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 18:40:34.775051 2429 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 18:40:34.775280 2429 mini_ranger.cc:66] Stopping Ranger...
I20260927 18:40:35.111651 2429 mini_ranger.cc:68] Stopped Ranger
2026-09-27 18:40:35.111 UTC [13353] LOG: received smart shutdown request
2026-09-27 18:40:35.115 UTC [13353] LOG: background worker "logical replication launcher" (PID 13362) exited with exit code 1
2026-09-27 18:40:35.116 UTC [13357] LOG: shutting down
2026-09-27 18:40:35.317 UTC [13353] LOG: database system is shut down
2026-09-27T18:40:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50485 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:40:35 dist-test-slave-9jll krb5kdc[14346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:40:35 dist-test-slave-9jll krb5kdc[14346](info): set up 2 sockets
Sep 27 18:40:35 dist-test-slave-9jll krb5kdc[14346](info): commencing operation
krb5kdc: starting...
W20260927 18:40:36.461963 2429 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:40:36 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:40:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:40:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.95.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.95.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/rangeradmin_127.2.95.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/rangeradmin_127.2.95.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.95.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.95.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/rangerlookup_127.2.95.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/rangerlookup_127.2.95.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.95.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.95.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/HTTP_127.2.95.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/HTTP_127.2.95.84@KRBTEST.COM.keytab.
I20260927 18:40:36.797750 2429 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknccDtC/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/postgres -l logfile start
2026-09-27 18:40:39.800 UTC [14376] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 18:40:39.800 UTC [14376] LOG: listening on IPv4 address "127.2.95.84", port 53419
2026-09-27 18:40:39.805 UTC [14376] LOG: listening on Unix socket "/tmp/.s.PGSQL.53419"
2026-09-27 18:40:39.826 UTC [14379] LOG: database system was shut down at 2026-09-27 18:40:37 UTC
2026-09-27 18:40:39.833 UTC [14376] LOG: database system is ready to accept connections
I20260927 18:40:40.725730 2429 mini_postgres.cc:96] Postgres bound to 53419
2026-09-27 18:40:40.746 UTC [14387] FATAL: database "slave" does not exist
127.2.95.84:53419 - accepting connections
I20260927 18:40:40.746994 2429 mini_ranger.cc:161] Starting Ranger...
I20260927 18:40:40.773293 2429 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 18:40:41.404844 2429 mini_ranger.cc:87] Created ranger Postgres database
I20260927 18:40:41.404944 2429 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-admin
2026-09-27 18:40:41,688 [I] DB FLAVOR :POSTGRES
2026-09-27 18:40:41,689 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:40:41,689 [I] Checking connection..
2026-09-27 18:40:41,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 18:40:41,957 [I] Checking connection passed.
2026-09-27 18:40:41,958 [I] --------- Verifying version history table ---------
2026-09-27 18:40:41,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:40:42,238 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:40:42,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:40:42,510 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 18:40:42,511 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 18:40:42,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 18:40:42.778 UTC [14451] WARNING: there is no transaction in progress
2026-09-27 18:40:42,784 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 18:40:42,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 18:40:43,068 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 18:40:43,068 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:40:43,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 18:40:43,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 18:40:43,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9jll.c.gcp-upstream.internal','N') ;"
2026-09-27 18:40:43,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:40:44,119 [I] Table x_portal_user does not exist in database ranger
2026-09-27 18:40:44,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:40:44,391 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 18:40:44,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:40:44,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:40:44,936 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 18:40:44,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 18:40:46.045 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.067 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.089 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.108 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.127 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.150 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.213 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.221 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.240 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.252 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.265 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.281 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.292 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.300 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.308 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:46.315 UTC [14559] WARNING: there is no transaction in progress
2026-09-27 18:40:47,177 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 18:40:47,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 18:40:47,421 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 18:40:47,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 18:40:47,664 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 18:40:47,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 18:40:47,921 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 18:40:47,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 18:40:48,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 18:40:48,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9jll.c.gcp-upstream.internal';"
2026-09-27 18:40:48,693 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 18:40:48,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 18:40:48,957 [I] DB_PATCHES have already been applied
I20260927 18:40:48.963992 2429 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasknccDtC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 18:40:48.964083 2429 mini_ranger.cc:193] Using host: 127.2.95.84
I20260927 18:40:48.977741 2429 mini_ranger.cc:227] Ranger admin URL: http://127.2.95.84:41117
Sep 27, 2026 6:40:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:40:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:40:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 6:40:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 6:40:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 6:40:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 6:40:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 18:40:51.063622 2429 mini_ranger.cc:160] Time spent starting Ranger: real 10.317s user 0.009s sys 0.038s
Sep 27, 2026 6:41:11 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 6:41:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 6:41:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 6:41:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 6:41:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 6:41:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 18:41:12.834385 2429 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.95.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.95.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/rangerkms_127.2.95.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.95.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/rangerkms_127.2.95.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.95.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.95.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.95.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/HTTP_127.2.95.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.95.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/HTTP_127.2.95.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 18:41:13 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534473, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 18:41:13.022799 2429 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 18:41:13.045550 2429 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 18:41:14.013056 2429 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 18:41:14.013182 2429 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-kms
I20260927 18:41:14.013221 2429 mini_ranger_kms.cc:227] Using postgres at 127.2.95.84:53419
2026-09-27 18:41:14,358 [I] DB FLAVOR :POSTGRES
2026-09-27 18:41:14,358 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 18:41:14,358 [I] Checking connection
2026-09-27 18:41:14,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 18:41:14,597 [I] connection success
2026-09-27 18:41:14,598 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 18:41:14,598 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 18:41:14,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 18:41:14,837 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 18:41:14,838 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 18:41:14,838 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 18:41:14,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.95.84:53419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 18:41:15,106 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 18:41:15.663204 2429 mini_ranger_kms.cc:310] Created kms service
I20260927 18:41:15.909927 2429 mini_ranger_kms.cc:326] Created kudu user
I20260927 18:41:15.960815 2429 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 18:41:15 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534473, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:41:16.414865 2429 mini_ranger_kms.cc:384] Added ranger policy
I20260927 18:41:16.415200 2429 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknccDtC/build/asan/bin/postgresql.jar:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknccDtC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 18:41:16.415241 2429 mini_ranger_kms.cc:240] Using host: 127.2.95.84
I20260927 18:41:16.433238 2429 mini_ranger_kms.cc:276] Ranger KMS PID: 14741
I20260927 18:41:16.433370 2429 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.95.84:58789
Sep 27, 2026 6:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 6:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 6:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 6:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 6:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 6:41:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 18:41:18.561396 2429 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.539s user 0.094s sys 0.266s
I20260927 18:41:18.561733 2429 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 18:41:18.561908 2429 mini_ranger_kms.cc:401] 127.2.95.84:58789/kms/v1/keys
Sep 27 18:41:18 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534478, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.95.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 18:41:19 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534478, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.95.84@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
Sep 27 18:41:19 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534478, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.95.84@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
Sep 27 18:41:22 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:41:22.125331 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33619
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:33619
--ranger_config_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.95.84:58789/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:41:22.545624 14791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:22.546077 14791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:22.546221 14791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:22.554513 14791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:41:22.554678 14791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:22.554736 14791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:22.554828 14791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:41:22.554893 14791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:41:22.566529 14791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.95.84:58789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:33619
--ranger_config_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:22.568341 14791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:22.570111 14791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:22.583966 14796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:22.583987 14797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:22.584422 14799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:22.585165 14791 server_base.cc:1034] running on GCE node
I20260927 18:41:22.586223 14791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:22.588277 14791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:22.589560 14791 hybrid_clock.cc:648] HybridClock initialized: now 1790534482589471 us; error 57 us; skew 500 ppm
I20260927 18:41:22.590055 14791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:22 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:22.601302 14791 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:41:22.603430 14791 webserver.cc:460] Webserver started at http://127.2.95.126:38775/ using document root <none> and password file <none>
I20260927 18:41:22.604292 14791 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:22.604409 14791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:22.604789 14791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:41:22 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:41:22.664970 14791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "acaae83dc3de4584aa0952eab25014b3"
format_stamp: "Formatted at 2026-09-27 18:41:22 on dist-test-slave-9jll"
server_key: "e637ca98700fd6854c8b1991c39dce6f"
server_key_iv: "13d61c18046ba2e47e85e60aa6b4c158"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:22.666199 14791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "acaae83dc3de4584aa0952eab25014b3"
format_stamp: "Formatted at 2026-09-27 18:41:22 on dist-test-slave-9jll"
server_key: "e637ca98700fd6854c8b1991c39dce6f"
server_key_iv: "13d61c18046ba2e47e85e60aa6b4c158"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:22.673890 14791 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 18:41:22.694712 14806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:22.696516 14791 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:41:22.696728 14791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "acaae83dc3de4584aa0952eab25014b3"
format_stamp: "Formatted at 2026-09-27 18:41:22 on dist-test-slave-9jll"
server_key: "e637ca98700fd6854c8b1991c39dce6f"
server_key_iv: "13d61c18046ba2e47e85e60aa6b4c158"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:22.696981 14791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:22.725507 14791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:22.730517 14791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:22.730938 14791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:22.751873 14791 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:33619
I20260927 18:41:22.751893 14857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:33619 every 8 connection(s)
I20260927 18:41:22.753599 14791 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:41:22.758561 14858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:22.761179 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 14791
I20260927 18:41:22.761394 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:41:22.762192 2429 external_mini_cluster.cc:1428] Setting key cc1de0b25a25fcaf66a133bbe9b7e445
I20260927 18:41:22.772975 14858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3: Bootstrap starting.
I20260927 18:41:22.777839 14858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3: Neither blocks nor log segments found. Creating new log.
Sep 27 18:41:22 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:22.779307 14858 log.cc:824] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:22.783259 14858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3: No bootstrap required, opened a new log
I20260927 18:41:22.788836 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:22.764752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.2.95.126:33619)
0927 18:41:22.765662 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:22.765712 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:22.765770 (+ 58us) server_negotiation.cc:407] Connection header received
0927 18:41:22.767663 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:22.767692 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:22.768240 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:22.768699 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:22.770831 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:22.772445 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:22.774394 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:22.775240 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:22.775413 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:22.780640 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:22.780651 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:22.780672 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:22.780716 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:22.784095 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:22.784901 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:22.784909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:22.784917 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:22.785082 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:22.785477 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:22.785484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:22.785486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:22.786179 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:22.786539 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:22.787045 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 18:41:22.787546 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":412,"threads_started":1}
I20260927 18:41:22.792949 14858 raft_consensus.cc:348] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaae83dc3de4584aa0952eab25014b3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33619 } }
I20260927 18:41:22.793489 14858 raft_consensus.cc:374] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:22.793591 14858 raft_consensus.cc:729] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acaae83dc3de4584aa0952eab25014b3, State: Initialized, Role: FOLLOWER
I20260927 18:41:22.794273 14858 consensus_queue.cc:260] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaae83dc3de4584aa0952eab25014b3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33619 } }
I20260927 18:41:22.794499 14858 raft_consensus.cc:388] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:41:22.794600 14858 raft_consensus.cc:482] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:41:22.794739 14858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:22.797709 14858 raft_consensus.cc:504] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaae83dc3de4584aa0952eab25014b3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33619 } }
I20260927 18:41:22.798306 14858 leader_election.cc:302] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acaae83dc3de4584aa0952eab25014b3; no voters:
I20260927 18:41:22.798939 14858 leader_election.cc:288] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:41:22.799086 14863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:41:22.799832 14863 raft_consensus.cc:686] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 1 LEADER]: Becoming Leader. State: Replica: acaae83dc3de4584aa0952eab25014b3, State: Running, Role: LEADER
I20260927 18:41:22.800444 14863 consensus_queue.cc:237] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaae83dc3de4584aa0952eab25014b3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33619 } }
I20260927 18:41:22.801465 14858 sys_catalog.cc:564] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:41:22.804633 14865 sys_catalog.cc:455] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acaae83dc3de4584aa0952eab25014b3. Latest consensus state: current_term: 1 leader_uuid: "acaae83dc3de4584aa0952eab25014b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaae83dc3de4584aa0952eab25014b3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33619 } } }
I20260927 18:41:22.805085 14865 sys_catalog.cc:458] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:22.805892 14864 sys_catalog.cc:455] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acaae83dc3de4584aa0952eab25014b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaae83dc3de4584aa0952eab25014b3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33619 } } }
I20260927 18:41:22.806306 14864 sys_catalog.cc:458] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:22.807940 14871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:41:22.814491 14871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:41:22.815616 14858 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:41:22.841985 14871 catalog_manager.cc:1347] Generated new cluster ID: 3bf619798d2b4208be951db89441293d
I20260927 18:41:22.842149 14871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:41:22.869714 14871 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:41:22.872452 14871 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:41:22.884719 14871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3: Generated new TSK 0
I20260927 18:41:22.886350 14871 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:41:23.743734 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.95.84:58789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:33619
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:41:24 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534484, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 18:41:24.259526 14903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:24.259945 14903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:24.260138 14903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:24.268949 14903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:24.269116 14903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:24.269304 14903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:41:24.281177 14903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.95.84:58789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:33619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:24.283135 14903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:24.284914 14903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:24.306080 14914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:24.306617 14913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:24.308931 14916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:24.309810 14903 server_base.cc:1034] running on GCE node
I20260927 18:41:24.310966 14903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:24.312794 14903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:24.314073 14903 hybrid_clock.cc:648] HybridClock initialized: now 1790534484313991 us; error 73 us; skew 500 ppm
I20260927 18:41:24.314596 14903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:24 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534484, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:24.319761 14903 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:41:24.331413 14903 webserver.cc:460] Webserver started at http://127.2.95.65:42567/ using document root <none> and password file <none>
I20260927 18:41:24.332958 14903 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:24.333170 14903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:24.333614 14903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:41:24 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534484, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:41:24.354908 14903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "7466136d94774e5a893fb841043be6b9"
format_stamp: "Formatted at 2026-09-27 18:41:24 on dist-test-slave-9jll"
server_key: "43c0370ab7f12fd7356990503606f6a5"
server_key_iv: "0f93ecc0893cbe19ed09458231fa218b"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:24.356079 14903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "7466136d94774e5a893fb841043be6b9"
format_stamp: "Formatted at 2026-09-27 18:41:24 on dist-test-slave-9jll"
server_key: "43c0370ab7f12fd7356990503606f6a5"
server_key_iv: "0f93ecc0893cbe19ed09458231fa218b"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:24.363292 14903 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 18:41:24.405028 14923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:24.407964 14903 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.007s sys 0.001s
I20260927 18:41:24.408411 14903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "7466136d94774e5a893fb841043be6b9"
format_stamp: "Formatted at 2026-09-27 18:41:24 on dist-test-slave-9jll"
server_key: "43c0370ab7f12fd7356990503606f6a5"
server_key_iv: "0f93ecc0893cbe19ed09458231fa218b"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:24.408883 14903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:24.475351 14903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:24.481777 14903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:24.482302 14903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:24.494319 14903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:24.508530 14903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:24.508780 14903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:24.509032 14903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:24.509147 14903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 18:41:24 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534484, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
Sep 27 18:41:25 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534484, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:41:25.237560 15036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:44637 every 8 connection(s)
I20260927 18:41:25.238106 14903 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:44637
I20260927 18:41:25.240793 14903 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:41:25.246675 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 14903
I20260927 18:41:25.246939 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:41:25.247884 2429 external_mini_cluster.cc:1428] Setting key 69ea1d209ddb05fd1f43ba7a1c2cdc8f
Sep 27 18:41:25 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534484, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:25.286553 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:25.244963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:47221 (local address 127.2.95.126:33619)
0927 18:41:25.265119 (+ 20156us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:25.265126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:25.265153 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:41:25.265238 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:25.265245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:25.265386 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:25.265529 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:25.268431 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:25.269772 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:25.271903 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:25.272665 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:25.272864 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:25.277626 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:25.277633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:25.277636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:25.277736 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:25.281396 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:25.282577 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:25.282584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:25.282587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:25.282774 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:25.283480 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:25.283486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:25.283489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:25.283916 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:25.284072 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:25.285877 (+ 1805us) server_negotiation.cc:299] Negotiation successful
0927 18:41:25.286141 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19970,"thread_start_us":19872,"threads_started":1}
I20260927 18:41:25.287508 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:25.245648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:33619 (local address 127.2.95.65:47221)
0927 18:41:25.259497 (+ 13849us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:25.259585 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:25.262551 (+ 2966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:25.266108 (+ 3557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:25.266125 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:25.266982 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:25.268088 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:25.268129 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:25.270034 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:25.270039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:25.271717 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:25.271732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:25.273001 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:25.273011 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:25.273328 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:25.274775 (+ 1447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:25.274804 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:25.277331 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:25.281623 (+ 4292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:25.281635 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:25.281663 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:25.282404 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:25.282913 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:25.282918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:25.282922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:25.283255 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:25.284153 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:25.284162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:25.284755 (+ 593us) client_negotiation.cc:769] Sending connection context
0927 18:41:25.285136 (+ 381us) client_negotiation.cc:240] Negotiation successful
0927 18:41:25.285855 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13393,"thread_start_us":11872,"threads_started":1}
I20260927 18:41:25.289553 15037 heartbeater.cc:347] Connected to a master server at 127.2.95.126:33619
I20260927 18:41:25.289947 15037 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:25.290812 15037 heartbeater.cc:511] Master 127.2.95.126:33619 requested a full tablet report, sending...
I20260927 18:41:25.294086 14823 ts_manager.cc:194] Registered new tserver with Master: 7466136d94774e5a893fb841043be6b9 (127.2.95.65:44637)
I20260927 18:41:25.296627 14823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:47221
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
W20260927 18:41:25.366109 2429 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.95.66: real 0.114s user 0.000s sys 0.025s
I20260927 18:41:25.367259 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.95.84:58789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:33619
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:41:25.826135 15044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:25.826496 15044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:25.826634 15044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:25.834964 15044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:25.835120 15044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:25.835278 15044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:41:25.846786 15044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.95.84:58789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:33619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:25.848605 15044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:25.850376 15044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:25.866899 15057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:25.866995 15056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:25.866899 15059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:25.867614 15044 server_base.cc:1034] running on GCE node
I20260927 18:41:25.868415 15044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:25.869704 15044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:25.870977 15044 hybrid_clock.cc:648] HybridClock initialized: now 1790534485870925 us; error 50 us; skew 500 ppm
I20260927 18:41:25.871402 15044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:25 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534485, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:25.875283 15044 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:41:25.877159 15044 webserver.cc:460] Webserver started at http://127.2.95.66:41641/ using document root <none> and password file <none>
I20260927 18:41:25.878077 15044 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:25.878202 15044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:25.878547 15044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:41:25 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534485, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:41:25.899737 15044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "727c21e52ada4204a786a82860ee481f"
format_stamp: "Formatted at 2026-09-27 18:41:25 on dist-test-slave-9jll"
server_key: "51e93122988559291457af1d36f3657f"
server_key_iv: "8d715245324e8a82810e20c58d50297b"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:25.900986 15044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "727c21e52ada4204a786a82860ee481f"
format_stamp: "Formatted at 2026-09-27 18:41:25 on dist-test-slave-9jll"
server_key: "51e93122988559291457af1d36f3657f"
server_key_iv: "8d715245324e8a82810e20c58d50297b"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:25.908159 15044 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260927 18:41:25.930038 15066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:25.932216 15044 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 18:41:25.932468 15044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "727c21e52ada4204a786a82860ee481f"
format_stamp: "Formatted at 2026-09-27 18:41:25 on dist-test-slave-9jll"
server_key: "51e93122988559291457af1d36f3657f"
server_key_iv: "8d715245324e8a82810e20c58d50297b"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:25.932740 15044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:25.955881 15044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:25.960912 15044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:25.961252 15044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:25.962456 15044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:25.964161 15044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:25.964246 15044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:25.964385 15044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:25.964437 15044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:26.000087 15044 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:39203
I20260927 18:41:26.000268 15178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:39203 every 8 connection(s)
I20260927 18:41:26.002395 15044 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:41:26.009848 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 15044
I20260927 18:41:26.010077 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:41:26.010756 2429 external_mini_cluster.cc:1428] Setting key 7bc31b08b2af73033e7d85371cd94f55
Sep 27 18:41:26 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534485, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:26.034101 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:26.006409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:35053 (local address 127.2.95.126:33619)
0927 18:41:26.007052 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:26.007058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:26.008782 (+ 1724us) server_negotiation.cc:407] Connection header received
0927 18:41:26.011852 (+ 3070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:26.011860 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:26.011980 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:26.012101 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:26.014347 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.016121 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.019468 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.020066 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.020236 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:26.025955 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:26.025964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:26.025969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:26.026026 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:26.029710 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:26.031000 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:26.031007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:26.031010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:26.031209 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:26.031793 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:26.031798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:26.031801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:26.032223 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:26.032398 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:26.033366 (+ 968us) server_negotiation.cc:299] Negotiation successful
0927 18:41:26.033661 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":359,"threads_started":1}
I20260927 18:41:26.035346 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:26.007048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:33619 (local address 127.2.95.66:35053)
0927 18:41:26.008514 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:26.008585 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:26.011375 (+ 2790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:26.012447 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:26.012461 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:26.013025 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:26.014096 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:26.014131 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.016380 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.016387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:26.019166 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:26.019195 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.020564 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.020573 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:26.020964 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:26.022000 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:26.022045 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:26.025431 (+ 3386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:26.030010 (+ 4579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:26.030023 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:26.030045 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:26.030797 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:26.031364 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:26.031370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:26.031372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:26.031666 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:26.032377 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:26.032386 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:26.032915 (+ 529us) client_negotiation.cc:769] Sending connection context
0927 18:41:26.033215 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 18:41:26.033853 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"thread_start_us":435,"threads_started":1}
I20260927 18:41:26.037235 15179 heartbeater.cc:347] Connected to a master server at 127.2.95.126:33619
I20260927 18:41:26.037626 15179 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:26.038628 15179 heartbeater.cc:511] Master 127.2.95.126:33619 requested a full tablet report, sending...
I20260927 18:41:26.040556 14823 ts_manager.cc:194] Registered new tserver with Master: 727c21e52ada4204a786a82860ee481f (127.2.95.66:39203)
I20260927 18:41:26.041550 14823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:35053
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:41:26.078158 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.95.84:58789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:33619
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:41:26.301272 15037 heartbeater.cc:503] Master 127.2.95.126:33619 was elected leader, sending a full tablet report...
W20260927 18:41:26.478024 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:26.478390 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:26.478492 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:26.486475 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:26.486626 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:26.486757 15185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:41:26.498694 15185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.95.84:58789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:33619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:26.500481 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:26.502345 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:26.517918 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:26.517902 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:26.518041 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:26.518563 15185 server_base.cc:1034] running on GCE node
I20260927 18:41:26.519433 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:26.520578 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:26.521804 15185 hybrid_clock.cc:648] HybridClock initialized: now 1790534486521736 us; error 50 us; skew 500 ppm
I20260927 18:41:26.522223 15185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:26 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:26.525949 15185 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:41:26.527801 15185 webserver.cc:460] Webserver started at http://127.2.95.67:43471/ using document root <none> and password file <none>
I20260927 18:41:26.528667 15185 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:26.528779 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:26.529081 15185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 18:41:26 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:41:26.549203 15185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "ff21ddeebc89471f84cb5d5a6463ec11"
format_stamp: "Formatted at 2026-09-27 18:41:26 on dist-test-slave-9jll"
server_key: "04afc6fa44fd54dcad6c39270e9434ed"
server_key_iv: "4a5baf0d6ca2bffa7b07b095c950fed2"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:26.550292 15185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "ff21ddeebc89471f84cb5d5a6463ec11"
format_stamp: "Formatted at 2026-09-27 18:41:26 on dist-test-slave-9jll"
server_key: "04afc6fa44fd54dcad6c39270e9434ed"
server_key_iv: "4a5baf0d6ca2bffa7b07b095c950fed2"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:26.557307 15185 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 18:41:26.577185 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:26.579346 15185 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:41:26.579582 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "ff21ddeebc89471f84cb5d5a6463ec11"
format_stamp: "Formatted at 2026-09-27 18:41:26 on dist-test-slave-9jll"
server_key: "04afc6fa44fd54dcad6c39270e9434ed"
server_key_iv: "4a5baf0d6ca2bffa7b07b095c950fed2"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:26.579864 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:26.610575 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:26.615440 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:26.615801 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:26.617051 15185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:26.618841 15185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:26.618924 15185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:26.619059 15185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:26.619113 15185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:26.655184 15185 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:36821
I20260927 18:41:26.655196 15312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:36821 every 8 connection(s)
I20260927 18:41:26.656770 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:41:26.661528 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 15185
I20260927 18:41:26.661768 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:41:26.662487 2429 external_mini_cluster.cc:1428] Setting key 2e85ecd06ed77ef68746130d24be1ec7
Sep 27 18:41:26 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534486, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:26.680330 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:26.660042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:45029 (local address 127.2.95.126:33619)
0927 18:41:26.660687 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:26.660693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:26.661557 (+ 864us) server_negotiation.cc:407] Connection header received
0927 18:41:26.664156 (+ 2599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:26.664162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:26.664315 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:26.664429 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:26.666517 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.667324 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.669132 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.669650 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.669808 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:26.673757 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:26.673763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:26.673766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:26.673809 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:26.676540 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:26.677382 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:26.677388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:26.677391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:26.677547 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:26.678070 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:26.678075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:26.678078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:26.678556 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:26.678727 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:26.679621 (+ 894us) server_negotiation.cc:299] Negotiation successful
0927 18:41:26.679923 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":328,"threads_started":1}
I20260927 18:41:26.681588 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:26.660394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:33619 (local address 127.2.95.67:45029)
0927 18:41:26.661304 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:26.661373 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:26.663766 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:26.664781 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:26.664799 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:26.665420 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:26.666330 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:26.666360 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.667490 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.667495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:26.668976 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:26.668990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.669925 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.669934 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:26.670222 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:26.671041 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:26.671070 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:26.673451 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:26.676713 (+ 3262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:26.676724 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:26.676748 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:26.677252 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:26.677655 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:26.677660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:26.677693 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:26.677931 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:26.678694 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:26.678704 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:26.679169 (+ 465us) client_negotiation.cc:769] Sending connection context
0927 18:41:26.679491 (+ 322us) client_negotiation.cc:240] Negotiation successful
0927 18:41:26.680144 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":324,"threads_started":1}
I20260927 18:41:26.683300 15313 heartbeater.cc:347] Connected to a master server at 127.2.95.126:33619
I20260927 18:41:26.683670 15313 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:26.684707 15313 heartbeater.cc:511] Master 127.2.95.126:33619 requested a full tablet report, sending...
I20260927 18:41:26.686502 14823 ts_manager.cc:194] Registered new tserver with Master: ff21ddeebc89471f84cb5d5a6463ec11 (127.2.95.67:36821)
I20260927 18:41:26.687439 14823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:45029
I20260927 18:41:26.692698 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:41:26.694839 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 14903
Sep 27 18:41:26 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:41:26.715026 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:26.696062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.2.95.65:44637)
0927 18:41:26.696783 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:26.696792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:26.696855 (+ 63us) server_negotiation.cc:407] Connection header received
0927 18:41:26.696929 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:26.696937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:26.697211 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:26.697366 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:26.699243 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.700438 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.702511 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.703454 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.703571 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:26.708380 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:26.708395 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:26.708417 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:26.708472 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:26.711647 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:26.712532 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:26.712540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:26.712543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:26.712702 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:26.713163 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:26.713168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:26.713170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:26.713707 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:26.714015 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:26.714368 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 18:41:26.714625 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":375,"threads_started":1}
I20260927 18:41:26.716842 15032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:26.809311 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 14903
I20260927 18:41:26.825134 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 15044
Sep 27 18:41:26 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:41:26.844457 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:26.826339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.2.95.66:39203)
0927 18:41:26.826939 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:26.826948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:26.826978 (+ 30us) server_negotiation.cc:407] Connection header received
0927 18:41:26.827051 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:26.827059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:26.827344 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:26.827513 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:26.829054 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.830245 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.831712 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.832405 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.832539 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:26.837630 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:26.837647 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:26.837719 (+ 72us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:26.837774 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:26.841036 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:26.842001 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:26.842010 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:26.842013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:26.842186 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:26.842597 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:26.842601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:26.842603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:26.843027 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:26.843379 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:26.843719 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 18:41:26.844048 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":304,"threads_started":1}
I20260927 18:41:26.846071 15174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:26.935290 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 15044
I20260927 18:41:26.950122 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 15185
Sep 27 18:41:26 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:41:26.970378 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:26.951351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.2.95.67:36821)
0927 18:41:26.952128 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:26.952137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:26.952177 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:41:26.952263 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:26.952271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:26.952636 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:26.952844 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:26.954452 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.955601 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.957062 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:26.957784 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:26.957910 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:26.963430 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:26.963449 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:26.963470 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:26.963518 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:26.967017 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:26.967909 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:26.967918 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:26.967921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:26.968081 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:26.968469 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:26.968473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:26.968475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:26.968922 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:26.969266 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:26.969616 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 18:41:26.969966 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":419,"threads_started":1}
I20260927 18:41:26.972193 15308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:27.064821 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 15185
I20260927 18:41:27.080092 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 14791
I20260927 18:41:27.081914 14853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:27.204408 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 14791
I20260927 18:41:27.226752 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33619
--webserver_interface=127.2.95.126
--webserver_port=38775
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:33619
--ranger_config_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.95.84:58789/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.95.84:58789/kms with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:41:27.622514 15325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:27.622920 15325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:27.623063 15325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:27.631126 15325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:41:27.631281 15325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:27.631366 15325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:27.631444 15325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:41:27.631513 15325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:41:27.642959 15325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.95.84:58789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:33619
--ranger_config_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=38775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:27.644719 15325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:27.646466 15325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:27.660032 15330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:27.660110 15331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:27.660063 15333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:27.661430 15325 server_base.cc:1034] running on GCE node
I20260927 18:41:27.662510 15325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:27.664363 15325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:27.665591 15325 hybrid_clock.cc:648] HybridClock initialized: now 1790534487665536 us; error 52 us; skew 500 ppm
I20260927 18:41:27.666064 15325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:27 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534487, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:27.679191 15325 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:41:27.681362 15325 webserver.cc:460] Webserver started at http://127.2.95.126:38775/ using document root <none> and password file <none>
I20260927 18:41:27.682348 15325 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:27.682502 15325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:41:27 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534487, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:41:27.759080 15325 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:41:27.762920 15341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:27.764631 15325 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:41:27.764858 15325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "acaae83dc3de4584aa0952eab25014b3"
format_stamp: "Formatted at 2026-09-27 18:41:22 on dist-test-slave-9jll"
server_key: "e637ca98700fd6854c8b1991c39dce6f"
server_key_iv: "13d61c18046ba2e47e85e60aa6b4c158"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:27.766085 15325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:27.794394 15325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:27.799928 15325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:27.800280 15325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:27.821806 15325 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:33619
I20260927 18:41:27.821873 15392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:33619 every 8 connection(s)
I20260927 18:41:27.823810 15325 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:41:27.830444 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 15325
I20260927 18:41:27.831629 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.95.84:58789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:44637
--local_ip_for_outbound_sockets=127.2.95.65
--tserver_master_addrs=127.2.95.126:33619
--webserver_port=42567
--webserver_interface=127.2.95.65
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.95.84:58789/kms with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:41:27.834020 15393 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:41:27.839686 15393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3: Bootstrap starting.
I20260927 18:41:27.864838 15393 log.cc:824] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:27.877502 15393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:41:27.878173 15393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3: Bootstrap complete.
I20260927 18:41:27.888365 15393 raft_consensus.cc:348] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaae83dc3de4584aa0952eab25014b3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33619 } }
I20260927 18:41:27.889372 15393 raft_consensus.cc:729] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: acaae83dc3de4584aa0952eab25014b3, State: Initialized, Role: FOLLOWER
I20260927 18:41:27.890244 15393 consensus_queue.cc:260] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaae83dc3de4584aa0952eab25014b3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33619 } }
I20260927 18:41:27.890514 15393 raft_consensus.cc:388] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:41:27.890645 15393 raft_consensus.cc:482] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:41:27.890820 15393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:41:27.895845 15393 raft_consensus.cc:504] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaae83dc3de4584aa0952eab25014b3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33619 } }
I20260927 18:41:27.896495 15393 leader_election.cc:302] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acaae83dc3de4584aa0952eab25014b3; no voters:
I20260927 18:41:27.897202 15393 leader_election.cc:288] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:41:27.897388 15397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:41:27.899400 15397 raft_consensus.cc:686] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [term 2 LEADER]: Becoming Leader. State: Replica: acaae83dc3de4584aa0952eab25014b3, State: Running, Role: LEADER
I20260927 18:41:27.899703 15393 sys_catalog.cc:564] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:41:27.900019 15397 consensus_queue.cc:237] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaae83dc3de4584aa0952eab25014b3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33619 } }
I20260927 18:41:27.904855 15399 sys_catalog.cc:455] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acaae83dc3de4584aa0952eab25014b3. Latest consensus state: current_term: 2 leader_uuid: "acaae83dc3de4584aa0952eab25014b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaae83dc3de4584aa0952eab25014b3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33619 } } }
I20260927 18:41:27.905323 15399 sys_catalog.cc:458] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:27.905975 15398 sys_catalog.cc:455] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "acaae83dc3de4584aa0952eab25014b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaae83dc3de4584aa0952eab25014b3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33619 } } }
I20260927 18:41:27.906306 15398 sys_catalog.cc:458] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:27.915571 15406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:41:27.922894 15406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:41:27.924064 15393 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 18:41:27.924633 15406 catalog_manager.cc:1259] Loaded cluster ID: 3bf619798d2b4208be951db89441293d
I20260927 18:41:27.924952 15406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:41:27.929795 15406 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:41:27.931720 15406 catalog_manager.cc:5879] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3: Loaded TSK: 0
I20260927 18:41:27.940351 15406 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:41:28.378808 15395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:28.379475 15395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:28.379706 15395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:28.392743 15395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:28.393045 15395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:28.393337 15395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:41:28.410897 15395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.95.84:58789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:44637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=42567
--tserver_master_addrs=127.2.95.126:33619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:28.413518 15395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:28.416504 15395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:28.462350 15431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:28.462651 15434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:28.463532 15430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:28.464663 15395 server_base.cc:1034] running on GCE node
I20260927 18:41:28.465910 15395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:28.472617 15395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:28.476130 15395 hybrid_clock.cc:648] HybridClock initialized: now 1790534488475982 us; error 137 us; skew 500 ppm
I20260927 18:41:28.476768 15395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:28 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534488, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:28.481980 15395 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:41:28.504145 15395 webserver.cc:460] Webserver started at http://127.2.95.65:42567/ using document root <none> and password file <none>
I20260927 18:41:28.506089 15395 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:28.506458 15395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:41:28 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534488, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:41:28.554883 15395 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.006s sys 0.001s
I20260927 18:41:28.560178 15442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:28.562348 15395 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 18:41:28.562589 15395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "7466136d94774e5a893fb841043be6b9"
format_stamp: "Formatted at 2026-09-27 18:41:24 on dist-test-slave-9jll"
server_key: "43c0370ab7f12fd7356990503606f6a5"
server_key_iv: "0f93ecc0893cbe19ed09458231fa218b"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:28.563997 15395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:28.600212 15395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:28.606431 15395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:28.606765 15395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:28.608180 15395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:28.610052 15395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:28.610205 15395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:28.610373 15395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:28.610474 15395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:28.653018 15395 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:44637
I20260927 18:41:28.653039 15554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:44637 every 8 connection(s)
I20260927 18:41:28.655714 15395 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:41:28.659490 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 15395
I20260927 18:41:28.660432 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.95.84:58789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:39203
--local_ip_for_outbound_sockets=127.2.95.66
--tserver_master_addrs=127.2.95.126:33619
--webserver_port=41641
--webserver_interface=127.2.95.66
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.95.84:58789/kms with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:41:28 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534488, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:28.703969 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:28.662225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:33619 (local address 127.2.95.65:59297)
0927 18:41:28.663993 (+ 1768us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:28.664053 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:28.666779 (+ 2726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:28.677444 (+ 10665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:28.677461 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:28.678326 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:28.679411 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:28.679449 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:28.682326 (+ 2877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:28.682334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:28.684704 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:28.684729 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:28.686704 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:28.686723 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:28.687120 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:28.688179 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:28.688217 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:28.691919 (+ 3702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:28.697383 (+ 5464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:28.697396 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:28.697419 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:28.698435 (+ 1016us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:28.699176 (+ 741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:28.699182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:28.699186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:28.699424 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:28.700473 (+ 1049us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:28.700484 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:28.701156 (+ 672us) client_negotiation.cc:769] Sending connection context
0927 18:41:28.701495 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 18:41:28.702280 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1379,"thread_start_us":1047,"threads_started":1}
I20260927 18:41:28.704046 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:28.663877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:59297 (local address 127.2.95.126:33619)
0927 18:41:28.675040 (+ 11163us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:28.675068 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:28.675148 (+ 80us) server_negotiation.cc:407] Connection header received
0927 18:41:28.675792 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:28.675825 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:28.676545 (+ 720us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:28.678270 (+ 1725us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:28.679756 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:28.682053 (+ 2297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:28.685008 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:28.686238 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:28.687839 (+ 1601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:28.692398 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:28.692414 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:28.692443 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:28.692512 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:28.697097 (+ 4585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:28.698730 (+ 1633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:28.698741 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:28.698750 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:28.699019 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:28.699578 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:28.699584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:28.699587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:28.700289 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:28.700516 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:28.701874 (+ 1358us) server_negotiation.cc:299] Negotiation successful
0927 18:41:28.702320 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10804,"thread_start_us":10512,"threads_started":1}
I20260927 18:41:28.708364 15555 heartbeater.cc:347] Connected to a master server at 127.2.95.126:33619
I20260927 18:41:28.708807 15555 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:28.709964 15555 heartbeater.cc:511] Master 127.2.95.126:33619 requested a full tablet report, sending...
W20260927 18:41:28.714080 15555 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:33619 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:41:28.714382 15555 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:28.714743 15555 heartbeater.cc:511] Master 127.2.95.126:33619 requested a full tablet report, sending...
I20260927 18:41:28.717963 15555 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:28.718389 15555 heartbeater.cc:511] Master 127.2.95.126:33619 requested a full tablet report, sending...
W20260927 18:41:28.720026 15555 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:41:29.170050 15559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:29.170518 15559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:29.170691 15559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:29.179716 15559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:29.179926 15559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:29.180191 15559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:41:29.198822 15559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.95.84:58789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:39203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=41641
--tserver_master_addrs=127.2.95.126:33619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:29.201606 15559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:29.204044 15559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:29.232618 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:29.233121 15578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:29.233727 15573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:29.234571 15559 server_base.cc:1034] running on GCE node
I20260927 18:41:29.235556 15559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:29.249342 15559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:29.252930 15559 hybrid_clock.cc:648] HybridClock initialized: now 1790534489252808 us; error 124 us; skew 500 ppm
I20260927 18:41:29.253618 15559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:29 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534489, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:29.259526 15559 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:41:29.261909 15559 webserver.cc:460] Webserver started at http://127.2.95.66:41641/ using document root <none> and password file <none>
I20260927 18:41:29.262871 15559 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:29.263042 15559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:41:29 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534489, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:41:29.309810 15559 fs_manager.cc:709] Time spent opening directory manager: real 0.022s user 0.005s sys 0.013s
I20260927 18:41:29.330642 15586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:29.333766 15559 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.008s sys 0.013s
I20260927 18:41:29.334168 15559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "727c21e52ada4204a786a82860ee481f"
format_stamp: "Formatted at 2026-09-27 18:41:25 on dist-test-slave-9jll"
server_key: "51e93122988559291457af1d36f3657f"
server_key_iv: "8d715245324e8a82810e20c58d50297b"
server_key_version: "kuduclusterkey@0"
Sep 27 18:41:29 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790534489, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:29.336170 15559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:29.395736 15559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:29.402623 15559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:29.403604 15559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:29.411635 15559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:29.418857 15559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:29.419026 15559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:29.419238 15559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:29.419332 15559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:29.691658 15559 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:39203
I20260927 18:41:29.691723 15699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:39203 every 8 connection(s)
I20260927 18:41:29.694489 15559 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:41:29.700320 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 15559
I20260927 18:41:29.701288 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.95.84:58789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:36821
--local_ip_for_outbound_sockets=127.2.95.67
--tserver_master_addrs=127.2.95.126:33619
--webserver_port=43471
--webserver_interface=127.2.95.67
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.95.84:58789/kms with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:41:29.720196 15555 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:29.720736 15555 heartbeater.cc:511] Master 127.2.95.126:33619 requested a full tablet report, sending...
I20260927 18:41:29.722868 15358 ts_manager.cc:194] Registered new tserver with Master: 7466136d94774e5a893fb841043be6b9 (127.2.95.65:44637)
I20260927 18:41:29.725270 15358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:59297
Sep 27 18:41:29 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534489, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:29.741438 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:29.698394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:41663 (local address 127.2.95.126:33619)
0927 18:41:29.713649 (+ 15255us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:29.713658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:29.713718 (+ 60us) server_negotiation.cc:407] Connection header received
0927 18:41:29.713812 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:29.713819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:29.713952 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:29.714121 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:29.717950 (+ 3829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:29.719201 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:29.722271 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:29.723121 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:29.723264 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:29.731916 (+ 8652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:29.731926 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:29.731931 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:29.731993 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:29.736393 (+ 4400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:29.737750 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:29.737757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:29.737760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:29.737951 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:29.738651 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:29.738658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:29.738661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:29.739118 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:29.739264 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:29.740577 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0927 18:41:29.740971 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15060,"thread_start_us":14819,"threads_started":1}
I20260927 18:41:29.744532 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:29.699257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:33619 (local address 127.2.95.66:41663)
0927 18:41:29.706686 (+ 7429us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:29.706782 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:29.709998 (+ 3216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:29.715618 (+ 5620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:29.715636 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:29.716448 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:29.717551 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:29.717590 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:29.719481 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:29.719487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:29.722031 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:29.722088 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:29.723704 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:29.723711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:29.724066 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:29.725007 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:29.725043 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:29.731462 (+ 6419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:29.736675 (+ 5213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:29.736689 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:29.736715 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:29.737515 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:29.738129 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:29.738134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:29.738138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:29.738498 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:29.739399 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:29.739410 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:29.740009 (+ 599us) client_negotiation.cc:769] Sending connection context
0927 18:41:29.741775 (+ 1766us) client_negotiation.cc:240] Negotiation successful
0927 18:41:29.742582 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6850,"thread_start_us":8819,"threads_started":1}
I20260927 18:41:29.756067 15700 heartbeater.cc:347] Connected to a master server at 127.2.95.126:33619
I20260927 18:41:29.756542 15700 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:29.757803 15700 heartbeater.cc:511] Master 127.2.95.126:33619 requested a full tablet report, sending...
I20260927 18:41:29.760560 15358 ts_manager.cc:194] Registered new tserver with Master: 727c21e52ada4204a786a82860ee481f (127.2.95.66:39203)
I20260927 18:41:29.762112 15358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:41663
Sep 27 18:41:30 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534489, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
W20260927 18:41:30.263242 15704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:30.263751 15704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:30.263944 15704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:30.275712 15704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:30.275915 15704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:30.276170 15704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:41:30.292155 15704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.95.84:58789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:36821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=43471
--tserver_master_addrs=127.2.95.126:33619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:30.294693 15704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:30.297034 15704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:30.319986 15709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:30.330832 15710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:30.333976 15712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:30.338142 15704 server_base.cc:1034] running on GCE node
I20260927 18:41:30.339488 15704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:30.347513 15704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:30.351096 15704 hybrid_clock.cc:648] HybridClock initialized: now 1790534490350975 us; error 114 us; skew 500 ppm
I20260927 18:41:30.351681 15704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:30 dist-test-slave-9jll krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534490, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:30.356953 15704 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:41:30.369797 15704 webserver.cc:460] Webserver started at http://127.2.95.67:43471/ using document root <none> and password file <none>
I20260927 18:41:30.371047 15704 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:30.371336 15704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 18:41:30 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534489, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
Sep 27 18:41:30 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534490, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for HTTP/127.2.95.84@KRBTEST.COM
I20260927 18:41:30.423197 15704 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.005s sys 0.001s
I20260927 18:41:30.428718 15721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:30.431726 15704 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260927 18:41:30.432148 15704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "ff21ddeebc89471f84cb5d5a6463ec11"
format_stamp: "Formatted at 2026-09-27 18:41:26 on dist-test-slave-9jll"
server_key: "04afc6fa44fd54dcad6c39270e9434ed"
server_key_iv: "4a5baf0d6ca2bffa7b07b095c950fed2"
server_key_version: "kuduclusterkey@0"
I20260927 18:41:30.434217 15704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:30.470585 15704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:30.477188 15704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:30.477643 15704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:30.485639 15704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:30.494700 15704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:30.494975 15704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 18:41:30.495165 15704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:30.495280 15704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:30.635385 15704 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:36821
I20260927 18:41:30.635861 15835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:36821 every 8 connection(s)
I20260927 18:41:30.638237 15704 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:41:30.642283 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 15704
Sep 27 18:41:30 dist-test-slave-9jll krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534490, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:30.666901 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.644240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.2.95.126:33619)
0927 18:41:30.644811 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:30.644818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:30.644842 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:41:30.644932 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:30.644938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:30.645041 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:30.645180 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:30.652909 (+ 7729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.654386 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.656770 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.657429 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.657535 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.660474 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:30.660479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:30.660484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:30.660547 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:30.663772 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:30.664530 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:30.664536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:30.664540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:30.664691 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:30.665089 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:30.665094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:30.665098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:30.665476 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:30.665818 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:30.666154 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 18:41:30.666454 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":415,"thread_start_us":273,"threads_started":1}
I20260927 18:41:30.670636 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.648348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:45899 (local address 127.2.95.126:33619)
0927 18:41:30.653052 (+ 4704us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:30.653059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:30.653081 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:41:30.653161 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:30.653167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:30.653281 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:30.653422 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:30.656122 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.657153 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.659176 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.659626 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.659740 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.663871 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:30.663877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:30.663881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:30.663924 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:30.666825 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:30.667686 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:30.667691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:30.667695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:30.667873 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:30.668445 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:30.668449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:30.668451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:30.668845 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:30.669003 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:30.670047 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0927 18:41:30.670261 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":4493,"thread_start_us":4310,"threads_started":1}
I20260927 18:41:30.671838 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.647957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:33619 (local address 127.2.95.67:45899)
0927 18:41:30.649123 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:30.649190 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:30.652029 (+ 2839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:30.654012 (+ 1983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:30.654028 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:30.654817 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:30.655884 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:30.655916 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.657378 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.657384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:30.659002 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:30.659017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.659890 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.659897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:30.660170 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.660882 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:30.660914 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:30.663364 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:30.666966 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:30.666978 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:30.667003 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:30.667566 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:30.668002 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:30.668008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:30.668012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:30.668307 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:30.668994 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:30.669005 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:30.669472 (+ 467us) client_negotiation.cc:769] Sending connection context
0927 18:41:30.669825 (+ 353us) client_negotiation.cc:240] Negotiation successful
0927 18:41:30.670485 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":471,"threads_started":1}
I20260927 18:41:30.673487 15837 heartbeater.cc:347] Connected to a master server at 127.2.95.126:33619
I20260927 18:41:30.673862 15837 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:30.674733 15837 heartbeater.cc:511] Master 127.2.95.126:33619 requested a full tablet report, sending...
I20260927 18:41:30.676618 15358 ts_manager.cc:194] Registered new tserver with Master: ff21ddeebc89471f84cb5d5a6463ec11 (127.2.95.67:36821)
I20260927 18:41:30.677609 15358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:45899
I20260927 18:41:30.683109 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:41:30.703383 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.686750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.2.95.126:33619)
0927 18:41:30.687000 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:30.687009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:30.687348 (+ 339us) server_negotiation.cc:407] Connection header received
0927 18:41:30.687519 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:30.687523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:30.687612 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:30.687706 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:30.689452 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.690312 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.692470 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.692941 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.693085 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.696733 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:30.696739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:30.696743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:30.696797 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:30.700010 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:30.700927 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:30.700932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:30.700934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:30.701093 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:30.701543 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:30.701549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:30.701551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:30.701996 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:30.702216 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:30.702764 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 18:41:30.703005 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:41:30.710026 15358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:41:30.719177 15358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:41:30.732137 15555 heartbeater.cc:503] Master 127.2.95.126:33619 was elected leader, sending a full tablet report...
I20260927 18:41:30.749963 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.739731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:36821 (local address 127.0.0.1:42606)
0927 18:41:30.740374 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:30.740405 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:30.740524 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:30.742048 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:30.742053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:30.742086 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:30.742593 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:30.742605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.745076 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.745081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:30.746958 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:30.746971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.748576 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.748582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:30.749223 (+ 641us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.749254 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:41:30.749335 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:41:30.749455 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":394,"thread_start_us":345,"threads_started":1}
I20260927 18:41:30.750049 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.738064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:44637 (local address 127.0.0.1:54744)
0927 18:41:30.739086 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:30.739139 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:30.739389 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:30.740676 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:30.740685 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:30.740718 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:30.741200 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:30.741216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.743522 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.743527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:30.746298 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:30.746314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.748563 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.748570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:30.748896 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.748968 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 18:41:30.749140 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 18:41:30.749391 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":26,"thread_start_us":472,"threads_started":1}
I20260927 18:41:30.750144 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.738195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.2.95.65:44637)
0927 18:41:30.738956 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:30.738964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:30.739355 (+ 391us) server_negotiation.cc:407] Connection header received
0927 18:41:30.739632 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:30.739642 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:30.739977 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:30.740271 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:30.741432 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.743343 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.746490 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.748234 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.748369 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.748718 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:30.749208 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 18:41:30.749427 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":367,"threads_started":1}
I20260927 18:41:30.751513 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.740119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.2.95.67:36821)
0927 18:41:30.741178 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:30.741188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:30.741229 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:41:30.741344 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:30.741351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:30.741626 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:30.742825 (+ 1199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:30.743171 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.744932 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.747154 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.748321 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.748819 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.749063 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:30.750912 (+ 1849us) server_negotiation.cc:299] Negotiation successful
0927 18:41:30.751121 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":703,"threads_started":1}
I20260927 18:41:30.751792 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.740858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.2.95.66:39203)
0927 18:41:30.741499 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:30.741508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:30.742822 (+ 1314us) server_negotiation.cc:407] Connection header received
0927 18:41:30.743102 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:30.743111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:30.743385 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:30.743569 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:30.744583 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.746218 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.748821 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.750310 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.750461 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.750691 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:30.751259 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 18:41:30.751447 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":298,"threads_started":1}
I20260927 18:41:30.753306 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.741485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:39203 (local address 127.0.0.1:48372)
0927 18:41:30.742696 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:30.742720 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:30.742923 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:30.743868 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:30.743872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:30.743897 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:30.744377 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:30.744390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.746396 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.746401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:30.748602 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:30.748616 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.750864 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.750871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:30.751046 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.751076 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:41:30.752899 (+ 1823us) client_negotiation.cc:240] Negotiation successful
0927 18:41:30.753030 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":896,"threads_started":1}
I20260927 18:41:30.755618 15490 tablet_service.cc:1467] Processing CreateTablet for tablet 830aa60fff504fcebe4f2e246c40e546 (DEFAULT_TABLE table=test-table [id=0b32ae6b8c0d40e2966a9cd1da9d29c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:41:30.755617 15769 tablet_service.cc:1467] Processing CreateTablet for tablet 830aa60fff504fcebe4f2e246c40e546 (DEFAULT_TABLE table=test-table [id=0b32ae6b8c0d40e2966a9cd1da9d29c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:41:30.756834 15635 tablet_service.cc:1467] Processing CreateTablet for tablet 830aa60fff504fcebe4f2e246c40e546 (DEFAULT_TABLE table=test-table [id=0b32ae6b8c0d40e2966a9cd1da9d29c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:41:30.757529 15490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830aa60fff504fcebe4f2e246c40e546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:30.757550 15769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830aa60fff504fcebe4f2e246c40e546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:30.758298 15635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830aa60fff504fcebe4f2e246c40e546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:30.766490 15700 heartbeater.cc:503] Master 127.2.95.126:33619 was elected leader, sending a full tablet report...
I20260927 18:41:30.774621 15863 tablet_bootstrap.cc:492] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f: Bootstrap starting.
I20260927 18:41:30.775563 15862 tablet_bootstrap.cc:492] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9: Bootstrap starting.
I20260927 18:41:30.777040 15864 tablet_bootstrap.cc:492] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11: Bootstrap starting.
I20260927 18:41:30.780241 15862 tablet_bootstrap.cc:654] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:30.780472 15864 tablet_bootstrap.cc:654] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:30.781338 15863 tablet_bootstrap.cc:654] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:30.782006 15864 log.cc:824] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:30.782114 15862 log.cc:824] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:30.782889 15863 log.cc:824] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:30.785928 15864 tablet_bootstrap.cc:492] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11: No bootstrap required, opened a new log
I20260927 18:41:30.786321 15864 ts_tablet_manager.cc:1397] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 18:41:30.788738 15863 tablet_bootstrap.cc:492] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f: No bootstrap required, opened a new log
I20260927 18:41:30.788865 15862 tablet_bootstrap.cc:492] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9: No bootstrap required, opened a new log
I20260927 18:41:30.789080 15863 ts_tablet_manager.cc:1397] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.008s
I20260927 18:41:30.789162 15862 ts_tablet_manager.cc:1397] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260927 18:41:30.798153 15863 raft_consensus.cc:348] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466136d94774e5a893fb841043be6b9" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 44637 } } peers { permanent_uuid: "ff21ddeebc89471f84cb5d5a6463ec11" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36821 } } peers { permanent_uuid: "727c21e52ada4204a786a82860ee481f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39203 } }
I20260927 18:41:30.798154 15864 raft_consensus.cc:348] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466136d94774e5a893fb841043be6b9" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 44637 } } peers { permanent_uuid: "ff21ddeebc89471f84cb5d5a6463ec11" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36821 } } peers { permanent_uuid: "727c21e52ada4204a786a82860ee481f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39203 } }
I20260927 18:41:30.798695 15864 raft_consensus.cc:374] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:30.798691 15863 raft_consensus.cc:374] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:30.798801 15864 raft_consensus.cc:729] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff21ddeebc89471f84cb5d5a6463ec11, State: Initialized, Role: FOLLOWER
I20260927 18:41:30.798879 15863 raft_consensus.cc:729] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727c21e52ada4204a786a82860ee481f, State: Initialized, Role: FOLLOWER
I20260927 18:41:30.799875 15863 consensus_queue.cc:260] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466136d94774e5a893fb841043be6b9" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 44637 } } peers { permanent_uuid: "ff21ddeebc89471f84cb5d5a6463ec11" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36821 } } peers { permanent_uuid: "727c21e52ada4204a786a82860ee481f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39203 } }
I20260927 18:41:30.799841 15864 consensus_queue.cc:260] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466136d94774e5a893fb841043be6b9" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 44637 } } peers { permanent_uuid: "ff21ddeebc89471f84cb5d5a6463ec11" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36821 } } peers { permanent_uuid: "727c21e52ada4204a786a82860ee481f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39203 } }
I20260927 18:41:30.800977 15862 raft_consensus.cc:348] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466136d94774e5a893fb841043be6b9" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 44637 } } peers { permanent_uuid: "ff21ddeebc89471f84cb5d5a6463ec11" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36821 } } peers { permanent_uuid: "727c21e52ada4204a786a82860ee481f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39203 } }
I20260927 18:41:30.801644 15862 raft_consensus.cc:374] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:30.801771 15862 raft_consensus.cc:729] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7466136d94774e5a893fb841043be6b9, State: Initialized, Role: FOLLOWER
I20260927 18:41:30.802269 15863 ts_tablet_manager.cc:1428] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260927 18:41:30.802698 15862 consensus_queue.cc:260] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466136d94774e5a893fb841043be6b9" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 44637 } } peers { permanent_uuid: "ff21ddeebc89471f84cb5d5a6463ec11" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36821 } } peers { permanent_uuid: "727c21e52ada4204a786a82860ee481f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39203 } }
I20260927 18:41:30.804143 15837 heartbeater.cc:503] Master 127.2.95.126:33619 was elected leader, sending a full tablet report...
I20260927 18:41:30.805079 15864 ts_tablet_manager.cc:1428] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11: Time spent starting tablet: real 0.019s user 0.017s sys 0.001s
I20260927 18:41:30.808856 15862 ts_tablet_manager.cc:1428] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9: Time spent starting tablet: real 0.020s user 0.015s sys 0.005s
I20260927 18:41:30.855427 15868 raft_consensus.cc:482] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:41:30.855829 15868 raft_consensus.cc:504] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466136d94774e5a893fb841043be6b9" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 44637 } } peers { permanent_uuid: "ff21ddeebc89471f84cb5d5a6463ec11" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36821 } } peers { permanent_uuid: "727c21e52ada4204a786a82860ee481f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39203 } }
I20260927 18:41:30.858078 15868 leader_election.cc:288] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7466136d94774e5a893fb841043be6b9 (127.2.95.65:44637), ff21ddeebc89471f84cb5d5a6463ec11 (127.2.95.67:36821)
I20260927 18:41:30.867076 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.859723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:50657 (local address 127.2.95.67:36821)
0927 18:41:30.859940 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:30.859947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:30.860125 (+ 178us) server_negotiation.cc:407] Connection header received
0927 18:41:30.860282 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:30.860288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:30.860384 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:30.860484 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:30.861232 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.862313 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.864591 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.865902 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.866048 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.866171 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:30.866603 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 18:41:30.866753 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 18:41:30.867046 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.858815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:50999 (local address 127.2.95.65:44637)
0927 18:41:30.859108 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:30.859118 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:30.859470 (+ 352us) server_negotiation.cc:407] Connection header received
0927 18:41:30.859773 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:30.859780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:30.859889 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:30.859966 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:30.860927 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.862087 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.864635 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.865944 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.866065 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.866172 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:30.866603 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 18:41:30.866753 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 18:41:30.867292 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.859409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:36821 (local address 127.2.95.66:50657)
0927 18:41:30.860015 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:30.860035 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:30.860157 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:30.860542 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:30.860555 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:30.860582 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:30.861052 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:30.861062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.862458 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.862462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:30.864369 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:30.864384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.866175 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.866182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:30.866378 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.866469 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 18:41:30.866551 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:41:30.866793 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":157,"thread_start_us":308,"threads_started":1}
I20260927 18:41:30.867350 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.858494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:44637 (local address 127.2.95.66:50999)
0927 18:41:30.859319 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:30.859341 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:30.859553 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:30.860202 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:30.860207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:30.860239 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:30.860759 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:30.860771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.862294 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.862300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:30.864318 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:30.864336 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.866188 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.866193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:30.866383 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.866441 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 18:41:30.866552 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 18:41:30.866794 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":110,"thread_start_us":422,"threads_started":1}
I20260927 18:41:30.868481 15510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830aa60fff504fcebe4f2e246c40e546" candidate_uuid: "727c21e52ada4204a786a82860ee481f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7466136d94774e5a893fb841043be6b9" is_pre_election: true
I20260927 18:41:30.868481 15789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830aa60fff504fcebe4f2e246c40e546" candidate_uuid: "727c21e52ada4204a786a82860ee481f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff21ddeebc89471f84cb5d5a6463ec11" is_pre_election: true
I20260927 18:41:30.869187 15510 raft_consensus.cc:2454] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 727c21e52ada4204a786a82860ee481f in term 0.
I20260927 18:41:30.869181 15789 raft_consensus.cc:2454] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 727c21e52ada4204a786a82860ee481f in term 0.
I20260927 18:41:30.870078 15588 leader_election.cc:302] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 727c21e52ada4204a786a82860ee481f, 7466136d94774e5a893fb841043be6b9; no voters:
I20260927 18:41:30.870672 15868 raft_consensus.cc:2781] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:41:30.870898 15868 raft_consensus.cc:482] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:41:30.871037 15868 raft_consensus.cc:3037] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:30.873608 15868 raft_consensus.cc:504] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466136d94774e5a893fb841043be6b9" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 44637 } } peers { permanent_uuid: "ff21ddeebc89471f84cb5d5a6463ec11" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36821 } } peers { permanent_uuid: "727c21e52ada4204a786a82860ee481f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39203 } }
I20260927 18:41:30.874318 15868 leader_election.cc:288] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [CANDIDATE]: Term 1 election: Requested vote from peers 7466136d94774e5a893fb841043be6b9 (127.2.95.65:44637), ff21ddeebc89471f84cb5d5a6463ec11 (127.2.95.67:36821)
I20260927 18:41:30.875053 15510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830aa60fff504fcebe4f2e246c40e546" candidate_uuid: "727c21e52ada4204a786a82860ee481f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7466136d94774e5a893fb841043be6b9"
I20260927 18:41:30.875224 15789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830aa60fff504fcebe4f2e246c40e546" candidate_uuid: "727c21e52ada4204a786a82860ee481f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff21ddeebc89471f84cb5d5a6463ec11"
I20260927 18:41:30.875316 15510 raft_consensus.cc:3037] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:30.875489 15789 raft_consensus.cc:3037] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:30.878013 15510 raft_consensus.cc:2454] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 727c21e52ada4204a786a82860ee481f in term 1.
I20260927 18:41:30.878402 15789 raft_consensus.cc:2454] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 727c21e52ada4204a786a82860ee481f in term 1.
I20260927 18:41:30.878670 15588 leader_election.cc:302] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 727c21e52ada4204a786a82860ee481f, 7466136d94774e5a893fb841043be6b9; no voters:
I20260927 18:41:30.879073 15868 raft_consensus.cc:2781] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:41:30.880477 15868 raft_consensus.cc:686] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 1 LEADER]: Becoming Leader. State: Replica: 727c21e52ada4204a786a82860ee481f, State: Running, Role: LEADER
I20260927 18:41:30.881052 15868 consensus_queue.cc:237] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466136d94774e5a893fb841043be6b9" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 44637 } } peers { permanent_uuid: "ff21ddeebc89471f84cb5d5a6463ec11" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36821 } } peers { permanent_uuid: "727c21e52ada4204a786a82860ee481f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39203 } }
I20260927 18:41:30.887326 15357 catalog_manager.cc:5495] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f reported cstate change: term changed from 0 to 1, leader changed from <none> to 727c21e52ada4204a786a82860ee481f (127.2.95.66). New cstate: current_term: 1 leader_uuid: "727c21e52ada4204a786a82860ee481f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466136d94774e5a893fb841043be6b9" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 44637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff21ddeebc89471f84cb5d5a6463ec11" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "727c21e52ada4204a786a82860ee481f" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39203 } health_report { overall_health: HEALTHY } } }
W20260927 18:41:30.892227 15839 tablet.cc:2369] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:41:30.915953 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.909475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.2.95.66:39203)
0927 18:41:30.909711 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:30.909719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:30.909742 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:41:30.909924 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:30.909928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:30.910048 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:30.910199 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:41:30.910629 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.911513 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.913362 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.913860 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.913965 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:30.914537 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:41:30.915310 (+ 773us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:41:30.915440 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:30.915557 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 18:41:30.915647 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 18:41:30.919715 15556 tablet.cc:2369] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:41:30.927783 15789 raft_consensus.cc:1261] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11 [term 1 FOLLOWER]: Refusing update from remote peer 727c21e52ada4204a786a82860ee481f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:41:30.928604 15510 raft_consensus.cc:1261] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9 [term 1 FOLLOWER]: Refusing update from remote peer 727c21e52ada4204a786a82860ee481f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:41:30.929232 15868 consensus_queue.cc:1035] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff21ddeebc89471f84cb5d5a6463ec11" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:41:30.929731 15873 consensus_queue.cc:1035] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7466136d94774e5a893fb841043be6b9" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 44637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:41:30.963680 15358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:41:30.964138 15358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:41:30.970009 15358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acaae83dc3de4584aa0952eab25014b3: Sending DeleteTablet for 3 replicas of tablet 830aa60fff504fcebe4f2e246c40e546
I20260927 18:41:30.971654 15490 tablet_service.cc:1514] Processing DeleteTablet for tablet 830aa60fff504fcebe4f2e246c40e546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:41:30 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:54744
I20260927 18:41:30.971704 15769 tablet_service.cc:1514] Processing DeleteTablet for tablet 830aa60fff504fcebe4f2e246c40e546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:41:30 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:42606
I20260927 18:41:30.971892 15635 tablet_service.cc:1514] Processing DeleteTablet for tablet 830aa60fff504fcebe4f2e246c40e546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:41:30 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:48372
I20260927 18:41:30.981884 15888 tablet_replica.cc:330] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f: stopping tablet replica
I20260927 18:41:30.982574 15888 raft_consensus.cc:2229] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:41:30.982722 15887 tablet_replica.cc:330] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9: stopping tablet replica
I20260927 18:41:30.983466 15887 raft_consensus.cc:2229] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:41:30.983515 15888 raft_consensus.cc:2258] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:41:30.983819 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 15395
I20260927 18:41:30.984293 15887 raft_consensus.cc:2258] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:41:30.986593 15886 tablet_replica.cc:330] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11: stopping tablet replica
I20260927 18:41:30.987213 15886 raft_consensus.cc:2229] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:41:30.987903 15886 raft_consensus.cc:2258] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:41:30.988238 15888 ts_tablet_manager.cc:1905] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:41:30.990391 15887 ts_tablet_manager.cc:1905] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:41:30.996169 15886 ts_tablet_manager.cc:1905] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:41:30.997189 15888 ts_tablet_manager.cc:1918] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:41:30.997417 15888 log.cc:1196] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/830aa60fff504fcebe4f2e246c40e546
I20260927 18:41:30.998077 15888 ts_tablet_manager.cc:1939] T 830aa60fff504fcebe4f2e246c40e546 P 727c21e52ada4204a786a82860ee481f: Deleting consensus metadata
I20260927 18:41:30.999167 15887 ts_tablet_manager.cc:1918] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:41:30.999491 15887 log.cc:1196] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/830aa60fff504fcebe4f2e246c40e546
I20260927 18:41:31.000130 15887 ts_tablet_manager.cc:1939] T 830aa60fff504fcebe4f2e246c40e546 P 7466136d94774e5a893fb841043be6b9: Deleting consensus metadata
I20260927 18:41:31.000597 15343 catalog_manager.cc:4841] TS 727c21e52ada4204a786a82860ee481f (127.2.95.66:39203): tablet 830aa60fff504fcebe4f2e246c40e546 (table test-table [id=0b32ae6b8c0d40e2966a9cd1da9d29c3]) successfully deleted
I20260927 18:41:31.002396 15343 catalog_manager.cc:4841] TS 7466136d94774e5a893fb841043be6b9 (127.2.95.65:44637): tablet 830aa60fff504fcebe4f2e246c40e546 (table test-table [id=0b32ae6b8c0d40e2966a9cd1da9d29c3]) successfully deleted
I20260927 18:41:31.007117 15886 ts_tablet_manager.cc:1918] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:41:31.007357 15886 log.cc:1196] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/830aa60fff504fcebe4f2e246c40e546
I20260927 18:41:31.008004 15886 ts_tablet_manager.cc:1939] T 830aa60fff504fcebe4f2e246c40e546 P ff21ddeebc89471f84cb5d5a6463ec11: Deleting consensus metadata
I20260927 18:41:31.010563 15343 catalog_manager.cc:4841] TS ff21ddeebc89471f84cb5d5a6463ec11 (127.2.95.67:36821): tablet 830aa60fff504fcebe4f2e246c40e546 (table test-table [id=0b32ae6b8c0d40e2966a9cd1da9d29c3]) successfully deleted
I20260927 18:41:31.013324 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:30.985781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.2.95.65:44637)
0927 18:41:30.985993 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:30.986000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:30.986023 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:41:30.986102 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:30.986107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:30.986214 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:30.986314 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:30.988509 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.989865 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.992066 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:30.992828 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:30.992974 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:31.004097 (+ 11123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:31.004118 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:31.004144 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:31.004199 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:31.008847 (+ 4648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:31.010479 (+ 1632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:31.010493 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:31.010496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:31.010732 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:31.011338 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:31.011343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:31.011345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:31.011903 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:31.012282 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:31.012645 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 18:41:31.012930 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:41:31.014236 15550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:31.115195 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 15395
I20260927 18:41:31.138700 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 15559
I20260927 18:41:31.158764 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:31.139689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.2.95.66:39203)
0927 18:41:31.139925 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:31.139931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:31.139958 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:41:31.140124 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:31.140130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:31.140285 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:31.140431 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:31.142398 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:31.143799 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:31.145518 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:31.146620 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:31.146790 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:31.150173 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:31.150201 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:31.150236 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:31.150316 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:31.155054 (+ 4738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:31.156055 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:31.156067 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:31.156071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:31.156305 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:31.156679 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:31.156684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:31.156687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:31.157281 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:31.157660 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:31.158029 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 18:41:31.158326 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 18:41:31.159613 15695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:31.288492 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 15559
I20260927 18:41:31.305917 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 15704
I20260927 18:41:31.321172 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:31.306956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.2.95.67:36821)
0927 18:41:31.307135 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:31.307141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:31.307161 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:41:31.307260 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:31.307266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:31.307353 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:31.307438 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:31.308900 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:31.309814 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:31.311309 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:31.311919 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:31.312065 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:31.314975 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:31.314993 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:31.315015 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:31.315057 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:31.318327 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:31.319120 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:31.319128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:31.319131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:31.319307 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:31.319657 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:31.319660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:31.319662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:31.320065 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:31.320365 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:31.320573 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 18:41:31.320832 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:41:31.322041 15829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:31.418545 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 15704
I20260927 18:41:31.433769 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 15325
I20260927 18:41:31.435367 15388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:31.563115 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 15325
I20260927 18:41:31.585198 2429 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 18:41:31.923089 2429 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 18:41:31.923358 2429 mini_ranger.cc:66] Stopping Ranger...
I20260927 18:41:32.271283 2429 mini_ranger.cc:68] Stopped Ranger
2026-09-27 18:41:32.271 UTC [14376] LOG: received smart shutdown request
2026-09-27 18:41:32.274 UTC [14376] LOG: background worker "logical replication launcher" (PID 14385) exited with exit code 1
2026-09-27 18:41:32.275 UTC [14380] LOG: shutting down
2026-09-27 18:41:32.501 UTC [14376] LOG: database system is shut down
2026-09-27T18:41:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57193 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:41:32 dist-test-slave-9jll krb5kdc[15901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:41:32 dist-test-slave-9jll krb5kdc[15901](info): set up 2 sockets
Sep 27 18:41:32 dist-test-slave-9jll krb5kdc[15901](info): commencing operation
krb5kdc: starting...
W20260927 18:41:33.678493 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:41:33 dist-test-slave-9jll krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:41:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:41:33.963055 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:40873
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:40873,127.2.95.125:39119,127.2.95.124:43665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:41:34.381585 15914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:34.382030 15914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:34.382184 15914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:34.390401 15914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:41:34.390561 15914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:34.390621 15914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:34.390709 15914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:41:34.390784 15914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:41:34.402253 15914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:40873,127.2.95.125:39119,127.2.95.124:43665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:40873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:34.404181 15914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:34.405915 15914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:34.421973 15919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:34.421922 15920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:34.421891 15922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:34.422919 15914 server_base.cc:1034] running on GCE node
I20260927 18:41:34.423975 15914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:34.425957 15914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:34.427218 15914 hybrid_clock.cc:648] HybridClock initialized: now 1790534494427164 us; error 47 us; skew 500 ppm
I20260927 18:41:34.427681 15914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:34 dist-test-slave-9jll krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534494, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:34.431665 15914 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:41:34.433586 15914 webserver.cc:460] Webserver started at http://127.2.95.126:44197/ using document root <none> and password file <none>
I20260927 18:41:34.434643 15914 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:34.434818 15914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:34.435258 15914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:41:34.439342 15914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "e72d6fae331446f9bf8f883eb4e1b45e"
format_stamp: "Formatted at 2026-09-27 18:41:34 on dist-test-slave-9jll"
server_key: "2d95f19923bdd7f40549e1a1b095cc02"
server_key_iv: "07ef46d203590add6e808ecf46ddbfad"
server_key_version: "encryptionkey@0"
I20260927 18:41:34.440469 15914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "e72d6fae331446f9bf8f883eb4e1b45e"
format_stamp: "Formatted at 2026-09-27 18:41:34 on dist-test-slave-9jll"
server_key: "2d95f19923bdd7f40549e1a1b095cc02"
server_key_iv: "07ef46d203590add6e808ecf46ddbfad"
server_key_version: "encryptionkey@0"
I20260927 18:41:34.448138 15914 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 18:41:34.453263 15929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:34.455140 15914 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:41:34.455410 15914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "e72d6fae331446f9bf8f883eb4e1b45e"
format_stamp: "Formatted at 2026-09-27 18:41:34 on dist-test-slave-9jll"
server_key: "2d95f19923bdd7f40549e1a1b095cc02"
server_key_iv: "07ef46d203590add6e808ecf46ddbfad"
server_key_version: "encryptionkey@0"
I20260927 18:41:34.455685 15914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:34.473070 15914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:34.478434 15914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:34.478720 15914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:34.499714 15914 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:40873
I20260927 18:41:34.499738 15980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:40873 every 8 connection(s)
I20260927 18:41:34.501451 15914 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:41:34.504333 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 15914
I20260927 18:41:34.504591 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:41:34.505417 2429 external_mini_cluster.cc:1428] Setting key 07bfdbb30997fdde2f63cb8b9abfe628
I20260927 18:41:34.506227 15981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:34.514508 15981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.95.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.125@KRBTEST.COM" created.
Sep 27 18:41:34 dist-test-slave-9jll krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534494, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:34.546494 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:34.519082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.2.95.126:40873)
0927 18:41:34.520109 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:34.520129 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:34.520688 (+ 559us) server_negotiation.cc:407] Connection header received
0927 18:41:34.523560 (+ 2872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:34.523567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:34.523882 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:34.524072 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:34.525813 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:34.527315 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:34.529608 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:34.530367 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:34.530540 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:34.536520 (+ 5980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:34.536533 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:34.536559 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:34.536610 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:34.540961 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:34.542008 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:34.542019 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:34.542022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:34.542277 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:34.542710 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:34.542715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:34.542718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:34.543665 (+ 947us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:34.543921 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:34.544625 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 18:41:34.545176 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":189,"server-negotiator.queue_time_us":716,"thread_start_us":425,"threads_started":1}
I20260927 18:41:34.546547 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:34.519082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:40873 (local address 127.0.0.1:34902)
0927 18:41:34.520472 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:34.520531 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:34.523059 (+ 2528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:34.524125 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:34.524133 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:34.525004 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:34.525562 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:34.525587 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:34.527494 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:34.527500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:34.529429 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:34.529448 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:34.530664 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:34.530674 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:34.530898 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:34.531726 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:34.531753 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:34.536168 (+ 4415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:34.541189 (+ 5021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:34.541203 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:34.541217 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:34.541836 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:34.542394 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:34.542398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:34.542400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:34.542598 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:34.543885 (+ 1287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:34.543893 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:34.544167 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 18:41:34.544429 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 18:41:34.544983 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"spinlock_wait_cycles":37632,"thread_start_us":826,"threads_started":1}
I20260927 18:41:34.550407 15981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:41:34.551681 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:34.551206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:43698)
0927 18:41:34.551399 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:34.551528 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 18:41:34.552021 15931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
I20260927 18:41:34.553908 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:34.553456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:43700)
0927 18:41:34.553645 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:34.553751 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 18:41:34.554101 15981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:39119: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.125.keytab.
Entry for principal kudu/127.2.95.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.125.keytab.
I20260927 18:41:34.571856 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.125:39119
--webserver_interface=127.2.95.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.125.keytab
--principal=kudu/127.2.95.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:40873,127.2.95.125:39119,127.2.95.124:43665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:41:34.603510 15981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } attempt: 1
I20260927 18:41:34.605054 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:34.604427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:43704)
0927 18:41:34.604740 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:34.604862 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 18:41:34.606487 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:34.605783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:43712)
0927 18:41:34.605989 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:34.606094 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 18:41:34.606801 15981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:39119: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
I20260927 18:41:34.675134 15981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } attempt: 2
I20260927 18:41:34.676862 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:34.676142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:43728)
0927 18:41:34.676493 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:34.676626 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 18:41:34.678311 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:34.677800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:43730)
0927 18:41:34.677969 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:34.678090 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 18:41:34.678540 15981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:39119: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
I20260927 18:41:34.769877 15981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } attempt: 3
I20260927 18:41:34.771552 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:34.770914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:43744)
0927 18:41:34.771186 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:34.771319 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 18:41:34.772876 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:34.772400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:43754)
0927 18:41:34.772565 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:34.772681 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:41:34.773123 15981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:39119: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
I20260927 18:41:34.916446 15981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } attempt: 4
I20260927 18:41:34.918206 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:34.917424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:43756)
0927 18:41:34.917849 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:34.917984 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260927 18:41:34.919590 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:34.919075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:43758)
0927 18:41:34.919245 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:34.919388 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 18:41:34.919844 15981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:39119: Network error: Client connection negotiation failed: client connection to 127.2.95.125:39119: connect: Connection refused (error 111)
W20260927 18:41:34.966393 15988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:34.966802 15988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:34.966918 15988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:34.975039 15988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:41:34.975183 15988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:34.975235 15988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:34.975296 15988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:41:34.975340 15988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:41:34.986697 15988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:40873,127.2.95.125:39119,127.2.95.124:43665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.125.keytab
--principal=kudu/127.2.95.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.125:39119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.95.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:34.988408 15988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:34.990090 15988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:35.004110 15994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:35.004154 15996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:35.004285 15993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:35.005065 15988 server_base.cc:1034] running on GCE node
I20260927 18:41:35.006600 15988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:35.008900 15988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:35.010159 15988 hybrid_clock.cc:648] HybridClock initialized: now 1790534495010089 us; error 66 us; skew 500 ppm
I20260927 18:41:35.010628 15988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534495, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:35.014505 15988 init.cc:377] Logged in from keytab as kudu/127.2.95.125@KRBTEST.COM (short username kudu)
I20260927 18:41:35.016700 15988 webserver.cc:460] Webserver started at http://127.2.95.125:42095/ using document root <none> and password file <none>
I20260927 18:41:35.017577 15988 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:35.017733 15988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:35.018108 15988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:41:35.022017 15988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/data/instance:
uuid: "fcf77f65149042b7ac33ebe0817a28cf"
format_stamp: "Formatted at 2026-09-27 18:41:35 on dist-test-slave-9jll"
server_key: "7df4e8566765597f9488c1bb43524d6c"
server_key_iv: "5504b0726f549cb87c64390bf28ba8a5"
server_key_version: "encryptionkey@0"
I20260927 18:41:35.023000 15988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/wal/instance:
uuid: "fcf77f65149042b7ac33ebe0817a28cf"
format_stamp: "Formatted at 2026-09-27 18:41:35 on dist-test-slave-9jll"
server_key: "7df4e8566765597f9488c1bb43524d6c"
server_key_iv: "5504b0726f549cb87c64390bf28ba8a5"
server_key_version: "encryptionkey@0"
I20260927 18:41:35.029569 15988 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 18:41:35.034062 16003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:35.036018 15988 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260927 18:41:35.036229 15988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/wal
uuid: "fcf77f65149042b7ac33ebe0817a28cf"
format_stamp: "Formatted at 2026-09-27 18:41:35 on dist-test-slave-9jll"
server_key: "7df4e8566765597f9488c1bb43524d6c"
server_key_iv: "5504b0726f549cb87c64390bf28ba8a5"
server_key_version: "encryptionkey@0"
I20260927 18:41:35.036464 15988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:35.059785 15988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:35.065194 15988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:35.065527 15988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:35.086217 15988 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.125:39119
I20260927 18:41:35.086225 16054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.125:39119 every 8 connection(s)
I20260927 18:41:35.087940 15988 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
I20260927 18:41:35.091135 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 15988
I20260927 18:41:35.091377 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-1/wal/instance
I20260927 18:41:35.092048 2429 external_mini_cluster.cc:1428] Setting key 57dec27c4d4f7355bea2eb9169786746
I20260927 18:41:35.093657 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:35.104245 16055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.95.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.124@KRBTEST.COM" created.
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534495, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.125@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:35.134140 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.107579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.2.95.126:40873)
0927 18:41:35.108268 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.108275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.109392 (+ 1117us) server_negotiation.cc:407] Connection header received
0927 18:41:35.112116 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.112121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.112216 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.112334 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.114889 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.115810 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.118439 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.118945 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.119066 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.125836 (+ 6770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.125847 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.125852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.125922 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.129889 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.131041 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.131048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.131051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.131234 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.131803 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.131808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.131811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.132246 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.132412 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.133419 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.133740 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":327,"threads_started":1}
I20260927 18:41:35.135865 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.108189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:40873 (local address 127.0.0.1:34908)
0927 18:41:35.109171 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.109233 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:35.111733 (+ 2500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:35.112713 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:35.112728 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:35.113607 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:35.114644 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.114672 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.116011 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.116018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.118256 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.118273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.119227 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.119235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.119607 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.120486 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:35.120522 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:35.125360 (+ 4838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:35.130127 (+ 4767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.130139 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.130161 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.130848 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.131392 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.131397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.131400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.131662 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.132447 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:35.132456 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:35.132982 (+ 526us) client_negotiation.cc:769] Sending connection context
0927 18:41:35.133299 (+ 317us) client_negotiation.cc:240] Negotiation successful
0927 18:41:35.133924 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":370,"threads_started":1}
I20260927 18:41:35.138218 16055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534495, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.125@KRBTEST.COM for kudu/127.2.95.125@KRBTEST.COM
I20260927 18:41:35.156647 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.139132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:43770)
0927 18:41:35.139369 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.139389 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:35.139489 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:35.141060 (+ 1571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:35.141066 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:35.141714 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:35.141929 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.141943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.143644 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.143650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.145016 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.145033 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.146168 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.146176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.146458 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.147208 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:35.147241 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:35.150230 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:35.153898 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.153902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.153905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.154305 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.154683 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.154688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.154692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.154844 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.155587 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:35.155591 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:35.155753 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 18:41:35.155903 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 18:41:35.156220 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 18:41:35.156747 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.139481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.2.95.125:39119)
0927 18:41:35.140426 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.140437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.140478 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:41:35.140536 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.140543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.140872 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.141043 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.142134 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.143459 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.145189 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.145967 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.146089 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.150504 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.150519 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.150540 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.150581 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.153727 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.154444 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.154454 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.154456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.154581 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.154946 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.154951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.154953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.155462 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.155737 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.156075 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.156345 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":672,"thread_start_us":497,"threads_started":1}
I20260927 18:41:35.159718 16055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.124.keytab.
Entry for principal kudu/127.2.95.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.124.keytab.
I20260927 18:41:35.161391 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.160695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:33944)
0927 18:41:35.160978 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.161151 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 18:41:35.161789 16007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
I20260927 18:41:35.162122 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.124:43665
--webserver_interface=127.2.95.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.124.keytab
--principal=kudu/127.2.95.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:42175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:40873,127.2.95.125:39119,127.2.95.124:43665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:41:35.171440 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.170954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:33950)
0927 18:41:35.171143 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.171260 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 18:41:35.171756 16055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:43665: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
I20260927 18:41:35.219209 15981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } attempt: 5
I20260927 18:41:35.221135 16055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } attempt: 1
I20260927 18:41:35.222539 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.221964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:33966)
0927 18:41:35.222170 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.222312 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 18:41:35.223728 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.223346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:33980)
0927 18:41:35.223504 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.223595 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 18:41:35.225400 16055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:43665: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534494, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for kudu/127.2.95.125@KRBTEST.COM
I20260927 18:41:35.237331 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.220280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:43776)
0927 18:41:35.220544 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.220568 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:35.220741 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:35.221461 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:35.221466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:35.222087 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:35.222340 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.222353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.223670 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.223676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.225078 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.225098 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.226088 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.226095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.226428 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.227214 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:35.227240 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:35.229960 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:35.234080 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.234086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.234089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.234590 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.235052 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.235057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.235059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.235235 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.236014 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:35.236020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:35.236270 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 18:41:35.236450 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 18:41:35.236914 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 18:41:35.237517 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.220299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.2.95.125:39119)
0927 18:41:35.220565 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.220572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.220656 (+ 84us) server_negotiation.cc:407] Connection header received
0927 18:41:35.220959 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.220965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.221084 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.221195 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.222496 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.223492 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.225259 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.225925 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.226036 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.230244 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.230252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.230257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.230308 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.233873 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.234725 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.234732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.234735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.234917 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.235351 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.235357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.235360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.235835 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.235997 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.236899 (+ 902us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.237204 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 18:41:35.239220 15981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:41:35.240799 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.240124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:33982)
0927 18:41:35.240413 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.240530 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 18:41:35.242012 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.241598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:33986)
0927 18:41:35.241765 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.241873 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 18:41:35.242280 15981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:43665: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
I20260927 18:41:35.293550 15981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } attempt: 1
I20260927 18:41:35.293684 16055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } attempt: 2
I20260927 18:41:35.295228 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.294524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:34002)
0927 18:41:35.294822 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.294966 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 18:41:35.295271 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.294687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:34016)
0927 18:41:35.294965 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.295073 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 18:41:35.296598 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.296135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:34044)
0927 18:41:35.296307 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.296417 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 18:41:35.296635 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.296202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:34028)
0927 18:41:35.296394 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.296483 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 18:41:35.296844 15981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:43665: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
W20260927 18:41:35.296973 16055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:43665: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
I20260927 18:41:35.365099 15981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } attempt: 2
I20260927 18:41:35.366729 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.365996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:34052)
0927 18:41:35.366282 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.366467 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 18:41:35.368120 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.367654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:34056)
0927 18:41:35.367834 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.367948 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 18:41:35.368422 15981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:43665: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
I20260927 18:41:35.388219 16055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } attempt: 3
I20260927 18:41:35.390246 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.389494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:34058)
0927 18:41:35.389881 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.390009 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 18:41:35.391592 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.391075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:34068)
0927 18:41:35.391339 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.391439 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260927 18:41:35.391846 16055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:43665: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
I20260927 18:41:35.474720 15981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } attempt: 3
I20260927 18:41:35.476457 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.475762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:38184)
0927 18:41:35.476038 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.476177 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 18:41:35.477907 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.477369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:38196)
0927 18:41:35.477606 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.477741 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260927 18:41:35.478149 15981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:43665: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
I20260927 18:41:35.535086 16055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } attempt: 4
I20260927 18:41:35.536453 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.535915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:38202)
0927 18:41:35.536144 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.536250 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 18:41:35.537655 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.537309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:38214)
0927 18:41:35.537453 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.537532 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 18:41:35.537979 16055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:43665: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
W20260927 18:41:35.565557 16063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:35.566085 16063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:35.566242 16063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:35.574501 16063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:41:35.574656 16063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:35.574733 16063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:35.574815 16063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:41:35.574884 16063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:41:35.586416 16063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:42175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:40873,127.2.95.125:39119,127.2.95.124:43665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.124.keytab
--principal=kudu/127.2.95.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.124:43665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.95.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:35.588222 16063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:35.589952 16063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:35.603945 16068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:35.603888 16071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:35.603945 16069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:35.605322 16063 server_base.cc:1034] running on GCE node
I20260927 18:41:35.606350 16063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:35.608210 16063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:35.609454 16063 hybrid_clock.cc:648] HybridClock initialized: now 1790534495609397 us; error 40 us; skew 500 ppm
I20260927 18:41:35.610014 16063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534495, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:35.613884 16063 init.cc:377] Logged in from keytab as kudu/127.2.95.124@KRBTEST.COM (short username kudu)
I20260927 18:41:35.615886 16063 webserver.cc:460] Webserver started at http://127.2.95.124:41785/ using document root <none> and password file <none>
I20260927 18:41:35.616781 16063 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:35.616904 16063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:35.617304 16063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:41:35.621441 16063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/data/instance:
uuid: "3442ce37f7bf4eda9548cbc802bac3d0"
format_stamp: "Formatted at 2026-09-27 18:41:35 on dist-test-slave-9jll"
server_key: "746fd67003d3cbed035f080f9e20f01f"
server_key_iv: "615829de72357b3dc5de012c7e6654cc"
server_key_version: "encryptionkey@0"
I20260927 18:41:35.622538 16063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/wal/instance:
uuid: "3442ce37f7bf4eda9548cbc802bac3d0"
format_stamp: "Formatted at 2026-09-27 18:41:35 on dist-test-slave-9jll"
server_key: "746fd67003d3cbed035f080f9e20f01f"
server_key_iv: "615829de72357b3dc5de012c7e6654cc"
server_key_version: "encryptionkey@0"
I20260927 18:41:35.628862 16063 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 18:41:35.633549 16078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:35.635444 16063 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260927 18:41:35.635648 16063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/wal
uuid: "3442ce37f7bf4eda9548cbc802bac3d0"
format_stamp: "Formatted at 2026-09-27 18:41:35 on dist-test-slave-9jll"
server_key: "746fd67003d3cbed035f080f9e20f01f"
server_key_iv: "615829de72357b3dc5de012c7e6654cc"
server_key_version: "encryptionkey@0"
I20260927 18:41:35.635883 16063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:35.654881 16063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:35.655380 15981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } attempt: 4
I20260927 18:41:35.656714 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.656211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:38222)
0927 18:41:35.656435 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.656537 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 18:41:35.657977 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.657556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:38230)
0927 18:41:35.657740 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.657828 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 18:41:35.658201 15981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:43665: Network error: Client connection negotiation failed: client connection to 127.2.95.124:43665: connect: Connection refused (error 111)
I20260927 18:41:35.660755 16063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:35.661116 16063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:35.680042 16063 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.124:43665
I20260927 18:41:35.680061 16129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.124:43665 every 8 connection(s)
I20260927 18:41:35.681787 16063 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
I20260927 18:41:35.686477 16130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:35.692174 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 16063
I20260927 18:41:35.692433 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790534284715420-2429-0/minicluster-data/master-2/wal/instance
I20260927 18:41:35.692978 2429 external_mini_cluster.cc:1428] Setting key 5e45fc5a29f9e1c729752225b40ada35
I20260927 18:41:35.695317 16130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534495, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.124@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.124@KRBTEST.COM
I20260927 18:41:35.715170 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.698642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.2.95.126:40873)
0927 18:41:35.699322 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.699329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.699908 (+ 579us) server_negotiation.cc:407] Connection header received
0927 18:41:35.700082 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.700088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.700193 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.700319 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.701876 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.702664 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.704216 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.704647 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.704782 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.708624 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.708631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.708635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.708673 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.711659 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.712508 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.712515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.712518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.712658 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.713081 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.713085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.713087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.713412 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.713558 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.714445 (+ 887us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.714707 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":349,"threads_started":1}
I20260927 18:41:35.716449 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.698650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:40873 (local address 127.0.0.1:52162)
0927 18:41:35.699722 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.699771 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:35.699948 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:35.700434 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:35.700440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:35.701129 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:35.701651 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.701693 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.702828 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.702832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.704070 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.704082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.704897 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.704904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.705043 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.705803 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:35.705837 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:35.708360 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:35.711799 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.711813 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.711834 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.712384 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.712764 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.712769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.712773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.712972 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.713551 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:35.713561 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:35.714023 (+ 462us) client_negotiation.cc:769] Sending connection context
0927 18:41:35.714324 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 18:41:35.714909 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":634,"threads_started":1}
I20260927 18:41:35.716470 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.695213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.2.95.124:43665)
0927 18:41:35.696178 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.696193 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.696250 (+ 57us) server_negotiation.cc:407] Connection header received
0927 18:41:35.698212 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.698251 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.698796 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.699272 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.701412 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.702743 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.704195 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.704780 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.704969 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.709469 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.709478 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.709494 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.709525 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.712692 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.713473 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.713490 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.713494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.713634 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.714098 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.714101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.714104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.714517 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.714850 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.715178 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.715450 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":444,"threads_started":1}
I20260927 18:41:35.718408 16130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534495, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.124@KRBTEST.COM for kudu/127.2.95.125@KRBTEST.COM
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:35.734433 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.719285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.2.95.125:39119)
0927 18:41:35.719522 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.719529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.719552 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:41:35.719625 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.719631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.719732 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.719841 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.720711 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.721629 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.723373 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.724040 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.724190 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.728047 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.728053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.728057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.728099 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.731358 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.732311 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.732316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.732318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.732496 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.732922 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.732926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.732928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.733252 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.733379 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.733897 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.734109 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 18:41:35.734522 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.719105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:39119 (local address 127.0.0.1:58904)
0927 18:41:35.719381 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.719397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:35.719484 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:35.719890 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:35.719895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:35.720370 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:35.720541 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.720550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.721858 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.721863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.723184 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.723209 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.724231 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.724236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.724433 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.725097 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:35.725120 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:35.727806 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:35.731523 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.731529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.731533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.732167 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.732618 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.732623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.732626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.732795 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.733375 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:35.733380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:35.733573 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 18:41:35.733782 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 18:41:35.734126 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260927 18:41:35.735659 16130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:41:35.738267 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.720173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.2.95.126:40873)
0927 18:41:35.720364 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.720370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.720388 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:41:35.720591 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.720597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.720705 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.720810 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.722276 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.723340 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.725125 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.725788 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.725917 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.730989 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.730995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.730999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.731040 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.734807 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.735750 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.735757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.735760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.735935 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.736424 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.736430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.736433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.736840 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.737200 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.737556 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.737906 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534495, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.124@KRBTEST.COM for kudu/127.2.95.124@KRBTEST.COM
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.125@KRBTEST.COM
I20260927 18:41:35.752406 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.736629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.2.95.124:43665)
0927 18:41:35.736847 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.736853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.737056 (+ 203us) server_negotiation.cc:407] Connection header received
0927 18:41:35.737209 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.737214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.737298 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.737435 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.738429 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.739418 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.740993 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.741900 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.742040 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.745940 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.745948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.745954 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.745993 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.749280 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.750044 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.750051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.750055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.750203 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.750684 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.750688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.750690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.751134 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.751313 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.751720 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.752116 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":63}
I20260927 18:41:35.752431 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.736621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:38252)
0927 18:41:35.736966 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.736986 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:35.737104 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:35.737461 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:35.737465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:35.738084 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:35.738280 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.738292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.739542 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.739547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.740838 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.740851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.742047 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.742052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.742215 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.742907 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:35.742926 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:35.745662 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:35.749428 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.749433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.749436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.749928 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.750343 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.750348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.750352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.750540 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.751300 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:35.751305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:35.751458 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 18:41:35.751579 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 18:41:35.751963 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260927 18:41:35.758780 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.740271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.2.95.125:39119)
0927 18:41:35.740492 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.740499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.740549 (+ 50us) server_negotiation.cc:407] Connection header received
0927 18:41:35.740702 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.740708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.740795 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.740912 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.743152 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.744287 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.745969 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.746445 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.746562 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.752078 (+ 5516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.752086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.752091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.752143 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.755720 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.756594 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.756601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.756605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.756797 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.757172 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.757177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.757181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.757604 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.757954 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.758146 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.758423 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 18:41:35.759694 16130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0: Bootstrap starting.
I20260927 18:41:35.764192 16130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:35.764930 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:41:35.765810 16130 log.cc:824] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:35.770107 16130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0: No bootstrap required, opened a new log
I20260927 18:41:35.783347 16130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } }
I20260927 18:41:35.784206 16130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:35.784425 16130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3442ce37f7bf4eda9548cbc802bac3d0, State: Initialized, Role: FOLLOWER
I20260927 18:41:35.785583 16130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } }
I20260927 18:41:35.794317 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.771903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.2.95.126:40873)
0927 18:41:35.772097 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.772103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.772588 (+ 485us) server_negotiation.cc:407] Connection header received
0927 18:41:35.772924 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.772933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.773033 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.773125 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.776095 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.777419 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.780294 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.781102 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.781235 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.786008 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.786017 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.786022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.786081 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.790278 (+ 4197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.791495 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.791501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.791505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.791688 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.792160 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.792166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.792170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.792645 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.792956 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.793609 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.793908 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:41:35.795884 16130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:41:35.796236 16144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } } }
I20260927 18:41:35.796941 16144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:41:35.799407 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.771749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.2.95.124:43665)
0927 18:41:35.772021 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.772029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.779098 (+ 7069us) server_negotiation.cc:407] Connection header received
0927 18:41:35.779726 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.779732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.779827 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.779901 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.782703 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.783781 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.786183 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.787125 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.787268 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.791565 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.791574 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.791579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.791638 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.795713 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.796591 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.796596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.796599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.796839 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.797322 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.797328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.797331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.797781 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.798061 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.798691 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.798949 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:41:35.803742 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.772984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.2.95.125:39119)
0927 18:41:35.776599 (+ 3615us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.776606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.784413 (+ 7807us) server_negotiation.cc:407] Connection header received
0927 18:41:35.785041 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.785049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.785160 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.785239 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.787169 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.788388 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.790663 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.791369 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.791470 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.795621 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.795630 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.795635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.795698 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.800107 (+ 4409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.801443 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.801449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.801452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.801632 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.802093 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.802099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.802103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.802437 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.802715 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.803114 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.803359 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3412}
W20260927 18:41:35.811931 16155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:41:35.812182 16155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:41:35.806738 16063 master_runner.cc:186] Error getting master registration for 127.2.95.126:40873: OK, instance_id { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" instance_seqno: 1790534494493736 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:41:35.816695 16063 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:41:35.837270 16055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } attempt: 5
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534495, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.125@KRBTEST.COM for kudu/127.2.95.124@KRBTEST.COM
I20260927 18:41:35.857986 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.838421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:38266)
0927 18:41:35.838719 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.838743 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:35.838887 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:35.839653 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:35.839658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:35.840318 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:35.840604 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.840619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.841966 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.841972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.843621 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.843643 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.844503 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.844510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.844737 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.845489 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:35.845517 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:35.849605 (+ 4088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:35.854489 (+ 4884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.854495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.854498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.855043 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.855651 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.855655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.855658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.855895 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.856668 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:35.856674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:35.856900 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 18:41:35.857089 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 18:41:35.857530 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260927 18:41:35.859508 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.838451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.2.95.124:43665)
0927 18:41:35.839132 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.839139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.839157 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:41:35.839216 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.839221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.839324 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.839426 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.840786 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.841830 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.843798 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.844326 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.844449 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.849834 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.849842 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.849848 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.849897 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.854226 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.855229 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.855236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.855240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.855485 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.856026 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.856032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.856036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.856500 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.856685 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.858706 (+ 2021us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.859045 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543}
I20260927 18:41:35.865152 16055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf: Bootstrap starting.
I20260927 18:41:35.869515 16055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:35.871249 16055 log.cc:824] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:35.875815 16055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf: No bootstrap required, opened a new log
I20260927 18:41:35.888507 16055 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } }
I20260927 18:41:35.889092 16055 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:35.889223 16055 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf77f65149042b7ac33ebe0817a28cf, State: Initialized, Role: FOLLOWER
I20260927 18:41:35.890132 16055 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } }
I20260927 18:41:35.891374 16158 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } } }
I20260927 18:41:35.891739 16158 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:41:35.893307 16055 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:41:35.908841 16169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:41:35.909428 16169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:41:35.902585 15988 master_runner.cc:186] Error getting master registration for 127.2.95.126:40873: OK, instance_id { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" instance_seqno: 1790534494493736 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:41:35.913420 15988 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 18:41:35.913689 16084 tablet.cc:2369] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:41:35.935417 15981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } attempt: 5
Sep 27 18:41:35 dist-test-slave-9jll krb5kdc[15901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534494, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for kudu/127.2.95.124@KRBTEST.COM
I20260927 18:41:35.960368 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.936163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:43665 (local address 127.0.0.1:38268)
0927 18:41:35.936526 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:35.936551 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:35.936676 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:35.937393 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:35.937400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:35.938045 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:35.938302 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.938317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.940065 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.940074 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.941923 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:35.941945 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.943134 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.943142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:35.943411 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.944297 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:35.944326 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:35.947883 (+ 3557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:35.956454 (+ 8571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.956462 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.956468 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.957168 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.957714 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:35.957721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:35.957725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:35.957968 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:35.958755 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:35.958761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:35.958964 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 18:41:35.959130 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 18:41:35.959523 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260927 18:41:35.960243 16144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:41:35.960742 16144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } }
I20260927 18:41:35.961213 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:35.936795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.2.95.124:43665)
0927 18:41:35.937024 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:35.937030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:35.937051 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:41:35.937117 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:35.937122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:35.937206 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:35.937323 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:35.938531 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.939746 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.942112 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:35.942906 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:35.943065 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:35.948268 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:35.948278 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:35.948284 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:35.948343 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:35.956243 (+ 7900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.957344 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.957352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.957356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.957542 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:35.958087 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:35.958091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:35.958094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:35.958555 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:35.958726 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:35.959290 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 18:41:35.959605 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 18:41:35.963215 16144 leader_election.cc:288] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e72d6fae331446f9bf8f883eb4e1b45e (127.2.95.126:40873), fcf77f65149042b7ac33ebe0817a28cf (127.2.95.125:39119)
I20260927 18:41:35.964345 16030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcf77f65149042b7ac33ebe0817a28cf" is_pre_election: true
I20260927 18:41:35.965122 16030 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3442ce37f7bf4eda9548cbc802bac3d0 in term 0.
I20260927 18:41:35.965984 16080 leader_election.cc:302] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3442ce37f7bf4eda9548cbc802bac3d0, fcf77f65149042b7ac33ebe0817a28cf; no voters:
I20260927 18:41:35.967180 16144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:41:35.967564 16144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:41:35.967890 16144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:35.968500 15981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e: Bootstrap starting.
I20260927 18:41:35.973804 15981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:35.975505 15981 log.cc:824] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:35.979842 16144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } }
I20260927 18:41:35.986217 16030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcf77f65149042b7ac33ebe0817a28cf"
I20260927 18:41:35.986474 15981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e: No bootstrap required, opened a new log
I20260927 18:41:35.986565 16030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:35.991482 16030 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3442ce37f7bf4eda9548cbc802bac3d0 in term 1.
I20260927 18:41:35.992524 16080 leader_election.cc:302] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3442ce37f7bf4eda9548cbc802bac3d0, fcf77f65149042b7ac33ebe0817a28cf; no voters:
I20260927 18:41:35.999789 15981 raft_consensus.cc:348] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } }
I20260927 18:41:36.000407 15981 raft_consensus.cc:374] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:35.999971 16144 leader_election.cc:288] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [CANDIDATE]: Term 1 election: Requested vote from peers e72d6fae331446f9bf8f883eb4e1b45e (127.2.95.126:40873), fcf77f65149042b7ac33ebe0817a28cf (127.2.95.125:39119)
I20260927 18:41:36.000551 15981 raft_consensus.cc:729] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72d6fae331446f9bf8f883eb4e1b45e, State: Initialized, Role: FOLLOWER
I20260927 18:41:36.001583 16144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:41:36.001695 15981 consensus_queue.cc:260] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } }
I20260927 18:41:36.002853 16144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [term 1 LEADER]: Becoming Leader. State: Replica: 3442ce37f7bf4eda9548cbc802bac3d0, State: Running, Role: LEADER
I20260927 18:41:36.014329 16144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } }
I20260927 18:41:36.018004 16173 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } } }
I20260927 18:41:36.018724 16173 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:41:36.019526 15981 sys_catalog.cc:564] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:41:36.019622 15955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72d6fae331446f9bf8f883eb4e1b45e"
I20260927 18:41:36.020289 15955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:36.023936 16172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3442ce37f7bf4eda9548cbc802bac3d0. Latest consensus state: current_term: 1 leader_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } } }
I20260927 18:41:36.024498 16172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:36.024993 15956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" is_pre_election: true
I20260927 18:41:36.030455 16176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:41:36.031446 15955 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3442ce37f7bf4eda9548cbc802bac3d0 in term 1.
I20260927 18:41:36.033174 16176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:41:36.052944 16030 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [term 1 FOLLOWER]: Refusing update from remote peer 3442ce37f7bf4eda9548cbc802bac3d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:41:36.054325 16172 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:41:36.063355 15955 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [term 1 FOLLOWER]: Refusing update from remote peer 3442ce37f7bf4eda9548cbc802bac3d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:41:36.064597 16172 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:41:36.069003 16009 tablet.cc:2369] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:41:36.087976 16173 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3442ce37f7bf4eda9548cbc802bac3d0. Latest consensus state: current_term: 1 leader_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } } }
I20260927 18:41:36.088471 16173 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:41:36.110522 16158 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3442ce37f7bf4eda9548cbc802bac3d0. Latest consensus state: current_term: 1 leader_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } } }
I20260927 18:41:36.110994 16158 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:41:36.111858 16173 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } } }
I20260927 18:41:36.112191 16173 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:41:36.116505 16181 mvcc.cc:205] Tried to move back new op lower bound from 7334029295811383296 to 7334029295698358272. Current Snapshot: MvccSnapshot[applied={T|T < 7334029295811383296}]
I20260927 18:41:36.121281 16172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } } }
I20260927 18:41:36.121654 16172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:36.123951 16144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } } }
I20260927 18:41:36.124336 16144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:36.124981 16195 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:41:36.127102 16176 catalog_manager.cc:1347] Generated new cluster ID: 78a9aed4ff23462180b17b8eda168ef2
I20260927 18:41:36.127224 16176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:41:36.138309 16158 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e72d6fae331446f9bf8f883eb4e1b45e" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40873 } } peers { permanent_uuid: "fcf77f65149042b7ac33ebe0817a28cf" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 39119 } } peers { permanent_uuid: "3442ce37f7bf4eda9548cbc802bac3d0" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 43665 } } }
I20260927 18:41:36.138748 16158 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:41:36.170979 16176 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:41:36.180217 16176 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:41:36.200980 16176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3442ce37f7bf4eda9548cbc802bac3d0: Generated new TSK 0
I20260927 18:41:36.202593 16176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:41:36.912032 16169 catalog_manager.cc:1259] Loaded cluster ID: 78a9aed4ff23462180b17b8eda168ef2
I20260927 18:41:36.912156 16169 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf: loading cluster ID for follower catalog manager: success
I20260927 18:41:36.915035 16169 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf: acquiring CA information for follower catalog manager: success
I20260927 18:41:36.917088 16169 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcf77f65149042b7ac33ebe0817a28cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:41:37.131075 16195 catalog_manager.cc:1259] Loaded cluster ID: 78a9aed4ff23462180b17b8eda168ef2
I20260927 18:41:37.131235 16195 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e: loading cluster ID for follower catalog manager: success
I20260927 18:41:37.134796 16195 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e: acquiring CA information for follower catalog manager: success
I20260927 18:41:37.137362 16195 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e72d6fae331446f9bf8f883eb4e1b45e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:41:37.326181 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 15914
I20260927 18:41:37.327944 15976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:37.452315 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 15914
I20260927 18:41:37.471206 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 15988
I20260927 18:41:37.472713 16050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:37.586673 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 15988
I20260927 18:41:37.608021 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 16063
I20260927 18:41:37.609530 16125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:37.731554 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 16063
2026-09-27T18:41:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5116 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:41:37 dist-test-slave-9jll krb5kdc[16206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:41:37 dist-test-slave-9jll krb5kdc[16206](info): set up 2 sockets
Sep 27 18:41:37 dist-test-slave-9jll krb5kdc[16206](info): commencing operation
krb5kdc: starting...
W20260927 18:41:38.794420 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:41:38 dist-test-slave-9jll krb5kdc[16206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:41:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:41:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:41:39.079068 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:38197
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:41:39.492857 16219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:39.493251 16219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:39.493359 16219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:39.501638 16219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:41:39.501819 16219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:39.501871 16219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:39.501933 16219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:41:39.501981 16219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:41:39.514358 16219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:38197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:39.516235 16219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:39.518044 16219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:39.533653 16224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:39.533579 16225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:39.533631 16227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:39.534648 16219 server_base.cc:1034] running on GCE node
I20260927 18:41:39.535885 16219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:39.538190 16219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:39.539491 16219 hybrid_clock.cc:648] HybridClock initialized: now 1790534499539430 us; error 57 us; skew 500 ppm
I20260927 18:41:39.540066 16219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:39 dist-test-slave-9jll krb5kdc[16206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534499, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:39.544672 16219 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:41:39.547101 16219 webserver.cc:460] Webserver started at http://127.2.95.126:40863/ using document root <none> and password file <none>
I20260927 18:41:39.548277 16219 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:39.548434 16219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:39.548821 16219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:41:39.552721 16219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "c01d938709044299a7a4e2a407e210f3"
format_stamp: "Formatted at 2026-09-27 18:41:39 on dist-test-slave-9jll"
server_key: "92f9e8c7380a10a9aad27ecf0e913c4f"
server_key_iv: "6ff339d987e52cd5e8a16cdc9498c57f"
server_key_version: "encryptionkey@0"
I20260927 18:41:39.553635 16219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "c01d938709044299a7a4e2a407e210f3"
format_stamp: "Formatted at 2026-09-27 18:41:39 on dist-test-slave-9jll"
server_key: "92f9e8c7380a10a9aad27ecf0e913c4f"
server_key_iv: "6ff339d987e52cd5e8a16cdc9498c57f"
server_key_version: "encryptionkey@0"
I20260927 18:41:39.560763 16219 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 18:41:39.566009 16234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:39.568112 16219 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:41:39.568370 16219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "c01d938709044299a7a4e2a407e210f3"
format_stamp: "Formatted at 2026-09-27 18:41:39 on dist-test-slave-9jll"
server_key: "92f9e8c7380a10a9aad27ecf0e913c4f"
server_key_iv: "6ff339d987e52cd5e8a16cdc9498c57f"
server_key_version: "encryptionkey@0"
I20260927 18:41:39.568648 16219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:39.585364 16219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:39.591181 16219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:39.591583 16219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:39.613641 16219 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:38197
I20260927 18:41:39.613657 16285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:38197 every 8 connection(s)
I20260927 18:41:39.615532 16219 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:41:39.620761 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 16219
I20260927 18:41:39.620939 16286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:39.621052 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:41:39.622002 2429 external_mini_cluster.cc:1428] Setting key b8d3c2ed12203a8380f854e524bb1665
I20260927 18:41:39.638978 16286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3: Bootstrap starting.
Sep 27 18:41:39 dist-test-slave-9jll krb5kdc[16206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:39.644207 16286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:39.645608 16286 log.cc:824] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:39.650068 16286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3: No bootstrap required, opened a new log
I20260927 18:41:39.650875 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:39.624317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.2.95.126:38197)
0927 18:41:39.625256 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:39.625273 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:39.625329 (+ 56us) server_negotiation.cc:407] Connection header received
0927 18:41:39.627256 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:39.627287 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:39.627829 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:39.628277 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:39.630580 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:39.632126 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:39.634103 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:39.635113 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:39.635296 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:39.641335 (+ 6039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:39.641349 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:39.641375 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:39.641442 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:39.645343 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:39.646625 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:39.646637 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:39.646652 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:39.647005 (+ 353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:39.647437 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:39.647442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:39.647445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:39.648133 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:39.648449 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:39.649022 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 18:41:39.649528 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":411,"threads_started":1}
I20260927 18:41:39.655827 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:41:39.660100 16286 raft_consensus.cc:348] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01d938709044299a7a4e2a407e210f3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38197 } }
I20260927 18:41:39.660709 16286 raft_consensus.cc:374] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:39.660822 16286 raft_consensus.cc:729] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01d938709044299a7a4e2a407e210f3, State: Initialized, Role: FOLLOWER
I20260927 18:41:39.661512 16286 consensus_queue.cc:260] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01d938709044299a7a4e2a407e210f3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38197 } }
I20260927 18:41:39.661850 16286 raft_consensus.cc:388] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:41:39.662005 16286 raft_consensus.cc:482] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:41:39.662180 16286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:39.664773 16286 raft_consensus.cc:504] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01d938709044299a7a4e2a407e210f3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38197 } }
I20260927 18:41:39.665441 16286 leader_election.cc:302] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c01d938709044299a7a4e2a407e210f3; no voters:
I20260927 18:41:39.666258 16286 leader_election.cc:288] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:41:39.666440 16291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:41:39.667187 16291 raft_consensus.cc:686] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [term 1 LEADER]: Becoming Leader. State: Replica: c01d938709044299a7a4e2a407e210f3, State: Running, Role: LEADER
I20260927 18:41:39.667834 16291 consensus_queue.cc:237] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01d938709044299a7a4e2a407e210f3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38197 } }
I20260927 18:41:39.668828 16286 sys_catalog.cc:564] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:41:39.671351 16292 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c01d938709044299a7a4e2a407e210f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01d938709044299a7a4e2a407e210f3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38197 } } }
I20260927 18:41:39.671864 16292 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:39.671485 16293 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c01d938709044299a7a4e2a407e210f3. Latest consensus state: current_term: 1 leader_uuid: "c01d938709044299a7a4e2a407e210f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c01d938709044299a7a4e2a407e210f3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 38197 } } }
I20260927 18:41:39.672307 16293 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:39.674163 16299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:41:39.680357 16299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:41:39.692544 16299 catalog_manager.cc:1347] Generated new cluster ID: 0c2e2085befa445e9b37909e8781387c
I20260927 18:41:39.692727 16299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:41:39.719551 16299 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:41:39.721566 16299 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:41:39.735740 16299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c01d938709044299a7a4e2a407e210f3: Generated new TSK 0
I20260927 18:41:39.737341 16299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:41:39.772780 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 16219
I20260927 18:41:39.774580 16281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:39.882951 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 16219
2026-09-27T18:41:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2148 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:41:40 dist-test-slave-9jll krb5kdc[16314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:41:40 dist-test-slave-9jll krb5kdc[16314](info): set up 2 sockets
Sep 27 18:41:40 dist-test-slave-9jll krb5kdc[16314](info): commencing operation
krb5kdc: starting...
W20260927 18:41:40.950513 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.006s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:41:41 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:41:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:41:41.230289 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:39751
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:39751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:41:41.637864 16327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:41.638345 16327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:41.638509 16327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:41.647176 16327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:41:41.647342 16327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:41.647406 16327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:41.647495 16327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:41:41.647552 16327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:41:41.659868 16327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:39751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:39751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:41.661793 16327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:41.663566 16327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:41.677340 16335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:41.677364 16333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:41.677403 16332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:41.678844 16327 server_base.cc:1034] running on GCE node
I20260927 18:41:41.679878 16327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:41.681845 16327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:41.683104 16327 hybrid_clock.cc:648] HybridClock initialized: now 1790534501683012 us; error 83 us; skew 500 ppm
I20260927 18:41:41.683574 16327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:41 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534501, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:41.687475 16327 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:41:41.689455 16327 webserver.cc:460] Webserver started at http://127.2.95.126:40849/ using document root <none> and password file <none>
I20260927 18:41:41.690431 16327 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:41.690557 16327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:41.690917 16327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:41:41.694744 16327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "1ba3f602018f4d85a4f645d2d3df0b95"
format_stamp: "Formatted at 2026-09-27 18:41:41 on dist-test-slave-9jll"
server_key: "cbbb43f18cf600fd56df06b582a39264"
server_key_iv: "d13b41c0e3f313b18fa9335a5ea3064e"
server_key_version: "encryptionkey@0"
I20260927 18:41:41.695647 16327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "1ba3f602018f4d85a4f645d2d3df0b95"
format_stamp: "Formatted at 2026-09-27 18:41:41 on dist-test-slave-9jll"
server_key: "cbbb43f18cf600fd56df06b582a39264"
server_key_iv: "d13b41c0e3f313b18fa9335a5ea3064e"
server_key_version: "encryptionkey@0"
I20260927 18:41:41.702378 16327 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 18:41:41.706763 16342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:41.708570 16327 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:41:41.708833 16327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "1ba3f602018f4d85a4f645d2d3df0b95"
format_stamp: "Formatted at 2026-09-27 18:41:41 on dist-test-slave-9jll"
server_key: "cbbb43f18cf600fd56df06b582a39264"
server_key_iv: "d13b41c0e3f313b18fa9335a5ea3064e"
server_key_version: "encryptionkey@0"
I20260927 18:41:41.709064 16327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:41.737104 16327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:41.743377 16327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:41.743736 16327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:41.765566 16327 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:39751
I20260927 18:41:41.765587 16393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:39751 every 8 connection(s)
I20260927 18:41:41.767498 16327 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:41:41.771248 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 16327
I20260927 18:41:41.771520 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:41:41.772275 2429 external_mini_cluster.cc:1428] Setting key e19169dba6dc2ad77cf52c9fa889b84e
I20260927 18:41:41.772552 16394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:41.789767 16394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Bootstrap starting.
Sep 27 18:41:41 dist-test-slave-9jll krb5kdc[16314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:41.794607 16394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:41.796177 16394 log.cc:824] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:41.800369 16394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: No bootstrap required, opened a new log
I20260927 18:41:41.801101 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:41.774720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.2.95.126:39751)
0927 18:41:41.775624 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:41.775640 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:41.775696 (+ 56us) server_negotiation.cc:407] Connection header received
0927 18:41:41.778521 (+ 2825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:41.778578 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:41.779461 (+ 883us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:41.780036 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:41.781722 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:41.783724 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:41.785302 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:41.786539 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:41.786752 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:41.791563 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:41.791576 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:41.791601 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:41.791667 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:41.795738 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:41.796631 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:41.796640 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:41.796651 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:41.796892 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:41.797394 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:41.797398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:41.797400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:41.798169 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:41.798515 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:41.798925 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 18:41:41.799523 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":374,"threads_started":1}
I20260927 18:41:41.809935 16394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } }
I20260927 18:41:41.810585 16394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:41.810720 16394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba3f602018f4d85a4f645d2d3df0b95, State: Initialized, Role: FOLLOWER
I20260927 18:41:41.811447 16394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } }
I20260927 18:41:41.811735 16394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:41:41.811894 16394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:41:41.812060 16394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:41.814776 16394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } }
I20260927 18:41:41.815443 16394 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba3f602018f4d85a4f645d2d3df0b95; no voters:
I20260927 18:41:41.816361 16394 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:41:41.816547 16399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:41:41.817435 16399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 1 LEADER]: Becoming Leader. State: Replica: 1ba3f602018f4d85a4f645d2d3df0b95, State: Running, Role: LEADER
I20260927 18:41:41.818114 16399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } }
I20260927 18:41:41.818939 16394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:41:41.821393 16400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } } }
I20260927 18:41:41.821369 16401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba3f602018f4d85a4f645d2d3df0b95. Latest consensus state: current_term: 1 leader_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } } }
I20260927 18:41:41.821866 16401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:41.821863 16400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:41.822896 16404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:41:41.831663 16404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:41:41.849009 16404 catalog_manager.cc:1347] Generated new cluster ID: 2524eb8f2d234e679199f2177dbef071
I20260927 18:41:41.849179 16404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:41:41.876682 16404 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:41:41.878494 16404 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:41:41.885324 16404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Generated new TSK 0
I20260927 18:41:41.886758 16404 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:41:41.962421 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39751
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:41:42.376832 16420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:42.377238 16420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:42.377362 16420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:42.385603 16420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:42.385790 16420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:42.385974 16420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:41:42.397257 16420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:42.399123 16420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:42.400830 16420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:42.417217 16426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:42.417168 16425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:42.417151 16428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:42.418063 16420 server_base.cc:1034] running on GCE node
I20260927 18:41:42.418879 16420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:42.420303 16420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:42.421553 16420 hybrid_clock.cc:648] HybridClock initialized: now 1790534502421493 us; error 46 us; skew 500 ppm
I20260927 18:41:42.421973 16420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:42 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534502, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:42.425964 16420 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:41:42.427846 16420 webserver.cc:460] Webserver started at http://127.2.95.65:41719/ using document root <none> and password file <none>
I20260927 18:41:42.428756 16420 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:42.428901 16420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:42.429256 16420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:41:42.433089 16420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "e5ae41da21564291b49e59721491b493"
format_stamp: "Formatted at 2026-09-27 18:41:42 on dist-test-slave-9jll"
server_key: "b1739478475041a6f55a94e732c04cc4"
server_key_iv: "f04af2d9f579c9f7b2dba8784945a60f"
server_key_version: "encryptionkey@0"
I20260927 18:41:42.434103 16420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "e5ae41da21564291b49e59721491b493"
format_stamp: "Formatted at 2026-09-27 18:41:42 on dist-test-slave-9jll"
server_key: "b1739478475041a6f55a94e732c04cc4"
server_key_iv: "f04af2d9f579c9f7b2dba8784945a60f"
server_key_version: "encryptionkey@0"
I20260927 18:41:42.440994 16420 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:41:42.445786 16435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:42.447832 16420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:41:42.448083 16420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "e5ae41da21564291b49e59721491b493"
format_stamp: "Formatted at 2026-09-27 18:41:42 on dist-test-slave-9jll"
server_key: "b1739478475041a6f55a94e732c04cc4"
server_key_iv: "f04af2d9f579c9f7b2dba8784945a60f"
server_key_version: "encryptionkey@0"
I20260927 18:41:42.448354 16420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:42.468137 16420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:42.473817 16420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:42.474167 16420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:42.475436 16420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:42.477147 16420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:42.477237 16420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:42.477377 16420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:42.477447 16420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:42.514532 16420 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:41223
I20260927 18:41:42.514569 16547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:41223 every 8 connection(s)
I20260927 18:41:42.516131 16420 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:41:42.521150 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 16420
I20260927 18:41:42.521404 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:41:42.522194 2429 external_mini_cluster.cc:1428] Setting key 9b59be526d7a6b8cdf70becd18ea66ee
Sep 27 18:41:42 dist-test-slave-9jll krb5kdc[16314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534502, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:42.542637 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:42.519235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:60717 (local address 127.2.95.126:39751)
0927 18:41:42.519853 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:42.519862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:42.521070 (+ 1208us) server_negotiation.cc:407] Connection header received
0927 18:41:42.523466 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:42.523473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:42.523586 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:42.523697 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:42.525879 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:42.526858 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:42.529012 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:42.529734 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:42.529980 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:42.534881 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:42.534889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:42.534894 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:42.534947 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:42.538515 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:42.539590 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:42.539595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:42.539599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:42.539768 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:42.540359 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:42.540364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:42.540367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:42.540771 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:42.540945 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:42.541944 (+ 999us) server_negotiation.cc:299] Negotiation successful
0927 18:41:42.542235 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":287,"threads_started":1}
I20260927 18:41:42.544062 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:42.519932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:60717)
0927 18:41:42.520832 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:42.520919 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:42.523122 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:42.524017 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:42.524028 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:42.524840 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:42.525635 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:42.525658 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:42.527133 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:42.527138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:42.528775 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:42.528798 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:42.530149 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:42.530157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:42.530687 (+ 530us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:42.531557 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:42.531592 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:42.534529 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:42.538814 (+ 4285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:42.538826 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:42.538852 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:42.539444 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:42.539959 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:42.539964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:42.539967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:42.540233 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:42.540939 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:42.540951 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:42.541469 (+ 518us) client_negotiation.cc:769] Sending connection context
0927 18:41:42.541878 (+ 409us) client_negotiation.cc:240] Negotiation successful
0927 18:41:42.542488 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":367,"threads_started":1}
I20260927 18:41:42.546725 16548 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39751
I20260927 18:41:42.547379 16548 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:42.548465 16548 heartbeater.cc:511] Master 127.2.95.126:39751 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
I20260927 18:41:42.551587 16359 ts_manager.cc:194] Registered new tserver with Master: e5ae41da21564291b49e59721491b493 (127.2.95.65:41223)
I20260927 18:41:42.553499 16359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:60717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:41:42.590178 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39751
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:41:43.009768 16554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:43.010198 16554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:43.010321 16554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:43.019176 16554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:43.019371 16554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:43.019542 16554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:41:43.031387 16554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:43.033221 16554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:43.035059 16554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:43.052346 16559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:43.052328 16562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:43.052392 16560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:43.055686 16554 server_base.cc:1034] running on GCE node
I20260927 18:41:43.056653 16554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:43.058107 16554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:43.059396 16554 hybrid_clock.cc:648] HybridClock initialized: now 1790534503059328 us; error 106 us; skew 500 ppm
I20260927 18:41:43.059940 16554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:43 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534503, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:43.071566 16554 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:41:43.074362 16554 webserver.cc:460] Webserver started at http://127.2.95.66:43227/ using document root <none> and password file <none>
I20260927 18:41:43.075606 16554 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:43.075780 16554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:43.076165 16554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:41:43.080178 16554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "8576c925fa0341efb839f8d5be77f89b"
format_stamp: "Formatted at 2026-09-27 18:41:43 on dist-test-slave-9jll"
server_key: "a93f51df1a4145ab19ec9c2c6f74d2cb"
server_key_iv: "79319f13fe6ebda0c2cef7de5f262e03"
server_key_version: "encryptionkey@0"
I20260927 18:41:43.081228 16554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "8576c925fa0341efb839f8d5be77f89b"
format_stamp: "Formatted at 2026-09-27 18:41:43 on dist-test-slave-9jll"
server_key: "a93f51df1a4145ab19ec9c2c6f74d2cb"
server_key_iv: "79319f13fe6ebda0c2cef7de5f262e03"
server_key_version: "encryptionkey@0"
I20260927 18:41:43.088410 16554 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 18:41:43.093454 16569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:43.095652 16554 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:41:43.095973 16554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "8576c925fa0341efb839f8d5be77f89b"
format_stamp: "Formatted at 2026-09-27 18:41:43 on dist-test-slave-9jll"
server_key: "a93f51df1a4145ab19ec9c2c6f74d2cb"
server_key_iv: "79319f13fe6ebda0c2cef7de5f262e03"
server_key_version: "encryptionkey@0"
I20260927 18:41:43.096238 16554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:43.110062 16554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:43.116387 16554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:43.116763 16554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:43.118140 16554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:43.120225 16554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:43.120333 16554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:43.120486 16554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:43.120543 16554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:43.159783 16554 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:43829
I20260927 18:41:43.159904 16681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:43829 every 8 connection(s)
I20260927 18:41:43.161312 16554 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:41:43.169188 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 16554
I20260927 18:41:43.169421 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:41:43.170106 2429 external_mini_cluster.cc:1428] Setting key 83157bf5306b6f8133c6b606455ef8e1
Sep 27 18:41:43 dist-test-slave-9jll krb5kdc[16314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534503, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:43.188758 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:43.164729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:54649 (local address 127.2.95.126:39751)
0927 18:41:43.165274 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:43.165282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:43.166409 (+ 1127us) server_negotiation.cc:407] Connection header received
0927 18:41:43.168621 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:43.168626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:43.168734 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:43.168852 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:43.171284 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.172255 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.174668 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.175209 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.175383 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:43.180738 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:43.180746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:43.180750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:43.180804 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:43.184384 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:43.185771 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:43.185777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:43.185781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:43.185986 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:43.186576 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:43.186582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:43.186585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:43.187015 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:43.187187 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:43.188064 (+ 877us) server_negotiation.cc:299] Negotiation successful
0927 18:41:43.188355 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":247,"threads_started":1}
I20260927 18:41:43.190208 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:43.165355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:54649)
0927 18:41:43.166194 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:43.166262 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:43.168248 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:43.169213 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:43.169228 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:43.170134 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:43.171086 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:43.171111 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.172433 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.172438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:43.174474 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:43.174492 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.175524 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.175533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:43.175871 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:43.176988 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:43.177028 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:43.180341 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:43.184630 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:43.184663 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:43.184722 (+ 59us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:43.185513 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:43.186122 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:43.186127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:43.186129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:43.186452 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:43.187148 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:43.187156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:43.187636 (+ 480us) client_negotiation.cc:769] Sending connection context
0927 18:41:43.187930 (+ 294us) client_negotiation.cc:240] Negotiation successful
0927 18:41:43.188477 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":348,"threads_started":1}
I20260927 18:41:43.192286 16682 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39751
I20260927 18:41:43.192698 16682 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:43.193651 16682 heartbeater.cc:511] Master 127.2.95.126:39751 requested a full tablet report, sending...
I20260927 18:41:43.195809 16359 ts_manager.cc:194] Registered new tserver with Master: 8576c925fa0341efb839f8d5be77f89b (127.2.95.66:43829)
I20260927 18:41:43.196843 16359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:54649
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:41:43.239295 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39751
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:41:43.559319 16548 heartbeater.cc:503] Master 127.2.95.126:39751 was elected leader, sending a full tablet report...
W20260927 18:41:43.670861 16688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:43.671284 16688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:43.671432 16688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:43.679684 16688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:43.679867 16688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:43.680052 16688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:41:43.691789 16688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:39751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:43.693660 16688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:43.695479 16688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:43.712935 16693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:43.712901 16694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:43.712960 16696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:43.713948 16688 server_base.cc:1034] running on GCE node
I20260927 18:41:43.714828 16688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:43.716058 16688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:43.717295 16688 hybrid_clock.cc:648] HybridClock initialized: now 1790534503717254 us; error 64 us; skew 500 ppm
I20260927 18:41:43.717732 16688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:43 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534503, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:43.722110 16688 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:41:43.724098 16688 webserver.cc:460] Webserver started at http://127.2.95.67:40041/ using document root <none> and password file <none>
I20260927 18:41:43.725003 16688 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:43.725123 16688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:43.725539 16688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:41:43.729434 16688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "f7dd6df161e044cf9ef045aab61a342d"
format_stamp: "Formatted at 2026-09-27 18:41:43 on dist-test-slave-9jll"
server_key: "fd889fe38f03e2b7c32ad6d9b9245417"
server_key_iv: "75b5e22fd1c0032d6083669a6c9ddbff"
server_key_version: "encryptionkey@0"
I20260927 18:41:43.730517 16688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "f7dd6df161e044cf9ef045aab61a342d"
format_stamp: "Formatted at 2026-09-27 18:41:43 on dist-test-slave-9jll"
server_key: "fd889fe38f03e2b7c32ad6d9b9245417"
server_key_iv: "75b5e22fd1c0032d6083669a6c9ddbff"
server_key_version: "encryptionkey@0"
I20260927 18:41:43.737885 16688 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 18:41:43.742940 16703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:43.745055 16688 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:41:43.745309 16688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "f7dd6df161e044cf9ef045aab61a342d"
format_stamp: "Formatted at 2026-09-27 18:41:43 on dist-test-slave-9jll"
server_key: "fd889fe38f03e2b7c32ad6d9b9245417"
server_key_iv: "75b5e22fd1c0032d6083669a6c9ddbff"
server_key_version: "encryptionkey@0"
I20260927 18:41:43.745597 16688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:43.780120 16688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:43.786827 16688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:43.787209 16688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:43.788511 16688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:43.790611 16688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:43.790702 16688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:43.790843 16688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:43.790902 16688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:43.828490 16688 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:45473
I20260927 18:41:43.828519 16815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:45473 every 8 connection(s)
I20260927 18:41:43.830281 16688 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:41:43.831828 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 16688
I20260927 18:41:43.832032 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:41:43.832634 2429 external_mini_cluster.cc:1428] Setting key d7a2b5c9a529c89de900fcf3930e7e3d
Sep 27 18:41:43 dist-test-slave-9jll krb5kdc[16314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534503, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:43.856494 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:43.834036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:34047 (local address 127.2.95.126:39751)
0927 18:41:43.834599 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:43.834603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:43.835751 (+ 1148us) server_negotiation.cc:407] Connection header received
0927 18:41:43.838161 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:43.838167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:43.838271 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:43.838379 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:43.840325 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.841292 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.843123 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.843618 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.843737 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:43.848459 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:43.848467 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:43.848471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:43.848525 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:43.852335 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:43.853336 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:43.853341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:43.853343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:43.853491 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:43.854038 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:43.854042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:43.854044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:43.854486 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:43.854671 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:43.855744 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0927 18:41:43.856045 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":261,"threads_started":1}
I20260927 18:41:43.857967 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:43.834569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:34047)
0927 18:41:43.835585 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:43.835634 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:43.837842 (+ 2208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:43.838682 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:43.838693 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:43.839392 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:43.840135 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:43.840155 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.841463 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.841467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:43.842977 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:43.842991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.843869 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.843876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:43.844138 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:43.845041 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:43.845074 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:43.848076 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:43.852543 (+ 4467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:43.852555 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:43.852576 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:43.853166 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:43.853593 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:43.853597 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:43.853600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:43.853897 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:43.854650 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:43.854661 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:43.855243 (+ 582us) client_negotiation.cc:769] Sending connection context
0927 18:41:43.855588 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 18:41:43.856320 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":367,"threads_started":1}
I20260927 18:41:43.860661 16816 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39751
I20260927 18:41:43.861114 16816 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:43.863117 16816 heartbeater.cc:511] Master 127.2.95.126:39751 requested a full tablet report, sending...
I20260927 18:41:43.865167 16359 ts_manager.cc:194] Registered new tserver with Master: f7dd6df161e044cf9ef045aab61a342d (127.2.95.67:45473)
I20260927 18:41:43.866205 16359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:34047
I20260927 18:41:43.877930 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:41:43.900947 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:43.883934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.2.95.126:39751)
0927 18:41:43.884221 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:43.884229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:43.884642 (+ 413us) server_negotiation.cc:407] Connection header received
0927 18:41:43.884885 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:43.884890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:43.884984 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:43.885095 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:43.887171 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.888098 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.889926 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.890497 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.890648 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:43.893833 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:43.893840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:43.893843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:43.893899 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:43.897513 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:43.898451 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:43.898456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:43.898459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:43.898615 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:43.899036 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:43.899040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:43.899042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:43.899499 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:43.899793 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:43.900109 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 18:41:43.900583 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 18:41:43.906736 16359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:41:43.916149 16359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:41:43.958923 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:43.934787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:45473 (local address 127.0.0.1:52018)
0927 18:41:43.949303 (+ 14516us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:43.949325 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:43.949537 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:43.950151 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:43.950157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:43.950181 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:43.950615 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:43.950626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.952709 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.952714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:43.955456 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:43.955479 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.957484 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.957490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:43.957744 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:43.957811 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 18:41:43.957975 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 18:41:43.958267 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14370,"mutex_wait_us":121,"spinlock_wait_cycles":1792,"thread_start_us":14176,"threads_started":1}
I20260927 18:41:43.959208 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:43.933315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.2.95.65:41223)
0927 18:41:43.948918 (+ 15603us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:43.948930 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:43.948968 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:41:43.949082 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:43.949090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:43.949331 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:43.950227 (+ 896us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:43.950755 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.952471 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.956134 (+ 3663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.957794 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.957945 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:43.958185 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:43.958463 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 18:41:43.958683 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15032,"thread_start_us":15170,"threads_started":1}
I20260927 18:41:43.958950 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:43.933141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:41223 (local address 127.0.0.1:38954)
0927 18:41:43.946266 (+ 13125us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:43.946288 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:43.946598 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:43.949599 (+ 3001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:43.949604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:43.949627 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:43.950114 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:43.950123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.952623 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.952629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:43.955893 (+ 3264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:43.955912 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.958025 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.958032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:43.958231 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:43.958263 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:41:43.958349 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 18:41:43.958498 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12957,"mutex_wait_us":144,"thread_start_us":13727,"threads_started":1}
I20260927 18:41:43.959715 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:43.933373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.2.95.66:43829)
0927 18:41:43.948205 (+ 14832us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:43.948218 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:43.948259 (+ 41us) server_negotiation.cc:407] Connection header received
0927 18:41:43.948409 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:43.948420 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:43.948807 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:43.949194 (+ 387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:43.950492 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.952588 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.956136 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.957826 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.957955 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:43.958148 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:43.959081 (+ 933us) server_negotiation.cc:299] Negotiation successful
0927 18:41:43.959288 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14633,"thread_start_us":14368,"threads_started":1}
I20260927 18:41:43.959951 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:43.933048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:43829 (local address 127.0.0.1:33012)
0927 18:41:43.945421 (+ 12373us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:43.945521 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:43.947466 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:43.949367 (+ 1901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:43.949377 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:43.949423 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:43.950109 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:43.950123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.952773 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.952782 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:43.955892 (+ 3110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:43.955912 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.958304 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.958309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:43.958566 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:43.958604 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:41:43.958681 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 18:41:43.958837 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11870,"mutex_wait_us":141,"thread_start_us":11607,"threads_started":1}
I20260927 18:41:43.960721 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:43.935683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.2.95.67:45473)
0927 18:41:43.948775 (+ 13092us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:43.948783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:43.949532 (+ 749us) server_negotiation.cc:407] Connection header received
0927 18:41:43.949749 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:43.949755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:43.949947 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:43.950129 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:43.950836 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.952512 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.955700 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:43.957258 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:43.957399 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:43.957727 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:43.959519 (+ 1792us) server_negotiation.cc:299] Negotiation successful
0927 18:41:43.959740 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12899,"thread_start_us":12722,"threads_started":1}
I20260927 18:41:43.964533 16617 tablet_service.cc:1467] Processing CreateTablet for tablet f4a2a32871c7486ca3cd50727a3a7300 (DEFAULT_TABLE table=test-table [id=196678b203e34bde8474b0ae66293e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:41:43.964542 16751 tablet_service.cc:1467] Processing CreateTablet for tablet f4a2a32871c7486ca3cd50727a3a7300 (DEFAULT_TABLE table=test-table [id=196678b203e34bde8474b0ae66293e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:41:43.964542 16483 tablet_service.cc:1467] Processing CreateTablet for tablet f4a2a32871c7486ca3cd50727a3a7300 (DEFAULT_TABLE table=test-table [id=196678b203e34bde8474b0ae66293e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:41:43.966336 16483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a2a32871c7486ca3cd50727a3a7300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:43.966337 16617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a2a32871c7486ca3cd50727a3a7300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:43.966338 16751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a2a32871c7486ca3cd50727a3a7300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:43.982604 16835 tablet_bootstrap.cc:492] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b: Bootstrap starting.
I20260927 18:41:43.986570 16836 tablet_bootstrap.cc:492] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d: Bootstrap starting.
I20260927 18:41:43.987030 16835 tablet_bootstrap.cc:654] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:43.988948 16835 log.cc:824] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:43.991109 16836 tablet_bootstrap.cc:654] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:43.992903 16836 log.cc:824] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:43.994647 16837 tablet_bootstrap.cc:492] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493: Bootstrap starting.
I20260927 18:41:43.999403 16837 tablet_bootstrap.cc:654] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:44.000101 16836 tablet_bootstrap.cc:492] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d: No bootstrap required, opened a new log
I20260927 18:41:44.000144 16835 tablet_bootstrap.cc:492] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b: No bootstrap required, opened a new log
I20260927 18:41:44.000443 16836 ts_tablet_manager.cc:1397] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260927 18:41:44.000485 16835 ts_tablet_manager.cc:1397] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20260927 18:41:44.001243 16837 log.cc:824] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:44.011075 16837 tablet_bootstrap.cc:492] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493: No bootstrap required, opened a new log
I20260927 18:41:44.010950 16836 raft_consensus.cc:348] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:44.011509 16837 ts_tablet_manager.cc:1397] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260927 18:41:44.011538 16836 raft_consensus.cc:374] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:44.011649 16836 raft_consensus.cc:729] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7dd6df161e044cf9ef045aab61a342d, State: Initialized, Role: FOLLOWER
I20260927 18:41:44.012708 16836 consensus_queue.cc:260] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:44.014374 16816 heartbeater.cc:503] Master 127.2.95.126:39751 was elected leader, sending a full tablet report...
I20260927 18:41:44.015321 16836 ts_tablet_manager.cc:1428] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260927 18:41:44.014503 16835 raft_consensus.cc:348] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:44.016426 16835 raft_consensus.cc:374] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:44.016599 16835 raft_consensus.cc:729] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8576c925fa0341efb839f8d5be77f89b, State: Initialized, Role: FOLLOWER
I20260927 18:41:44.017473 16835 consensus_queue.cc:260] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:44.019001 16682 heartbeater.cc:503] Master 127.2.95.126:39751 was elected leader, sending a full tablet report...
I20260927 18:41:44.019886 16835 ts_tablet_manager.cc:1428] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b: Time spent starting tablet: real 0.019s user 0.014s sys 0.005s
I20260927 18:41:44.024546 16837 raft_consensus.cc:348] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:44.025086 16837 raft_consensus.cc:374] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:44.025195 16837 raft_consensus.cc:729] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ae41da21564291b49e59721491b493, State: Initialized, Role: FOLLOWER
I20260927 18:41:44.025947 16837 consensus_queue.cc:260] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:44.028129 16837 ts_tablet_manager.cc:1428] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260927 18:41:44.056293 16841 raft_consensus.cc:482] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:41:44.056701 16841 raft_consensus.cc:504] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:44.059187 16841 leader_election.cc:288] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5ae41da21564291b49e59721491b493 (127.2.95.65:41223), 8576c925fa0341efb839f8d5be77f89b (127.2.95.66:43829)
I20260927 18:41:44.060485 16843 raft_consensus.cc:482] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:41:44.060890 16843 raft_consensus.cc:504] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:44.067694 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:44.059581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:41223 (local address 127.2.95.67:55355)
0927 18:41:44.060070 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:44.060094 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:44.060369 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:44.061053 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:44.061058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:44.061083 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:44.061272 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:44.061285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.062663 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.062668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:44.064650 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:44.064670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.066734 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.066740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:44.066996 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:44.067037 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:41:44.067150 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 18:41:44.067328 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321}
I20260927 18:41:44.068403 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:44.060043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:49315 (local address 127.2.95.66:43829)
0927 18:41:44.060256 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:44.060262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:44.060847 (+ 585us) server_negotiation.cc:407] Connection header received
0927 18:41:44.061159 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:44.061166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:44.061280 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:44.061374 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:44.062071 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.063171 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.065720 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.067114 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.067235 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:44.067344 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:44.067942 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 18:41:44.068106 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 18:41:44.068554 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:44.060020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:55355 (local address 127.2.95.65:41223)
0927 18:41:44.060232 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:44.060238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:44.060264 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:41:44.060635 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:44.060641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:44.060743 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:44.060848 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:44.061461 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.062523 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.064896 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.066331 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.066486 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:44.066622 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:44.068106 (+ 1484us) server_negotiation.cc:299] Negotiation successful
0927 18:41:44.068288 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:41:44.069254 16843 leader_election.cc:288] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8576c925fa0341efb839f8d5be77f89b (127.2.95.66:43829), f7dd6df161e044cf9ef045aab61a342d (127.2.95.67:45473)
I20260927 18:41:44.069446 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:44.059703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:43829 (local address 127.2.95.67:49315)
0927 18:41:44.060702 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:44.060722 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:44.060970 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:44.061411 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:44.061414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:44.061442 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:44.061882 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:44.061898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.063370 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.063376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:44.065484 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:44.065500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.067546 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.067552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:44.067734 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:44.067768 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:41:44.068901 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0927 18:41:44.069087 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":590,"threads_started":1}
I20260927 18:41:44.070982 16637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a2a32871c7486ca3cd50727a3a7300" candidate_uuid: "f7dd6df161e044cf9ef045aab61a342d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8576c925fa0341efb839f8d5be77f89b" is_pre_election: true
I20260927 18:41:44.071574 16503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a2a32871c7486ca3cd50727a3a7300" candidate_uuid: "f7dd6df161e044cf9ef045aab61a342d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ae41da21564291b49e59721491b493" is_pre_election: true
I20260927 18:41:44.072052 16637 raft_consensus.cc:2454] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7dd6df161e044cf9ef045aab61a342d in term 0.
I20260927 18:41:44.072065 16503 raft_consensus.cc:2454] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7dd6df161e044cf9ef045aab61a342d in term 0.
I20260927 18:41:44.073076 16707 leader_election.cc:302] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8576c925fa0341efb839f8d5be77f89b, f7dd6df161e044cf9ef045aab61a342d; no voters:
I20260927 18:41:44.073737 16841 raft_consensus.cc:2781] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:41:44.073938 16841 raft_consensus.cc:482] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:41:44.074045 16841 raft_consensus.cc:3037] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:44.077406 16841 raft_consensus.cc:504] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:44.079137 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:44.064515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:40271 (local address 127.2.95.66:43829)
0927 18:41:44.069106 (+ 4591us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:44.069111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:44.070913 (+ 1802us) server_negotiation.cc:407] Connection header received
0927 18:41:44.071169 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:44.071175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:44.071600 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:44.071690 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:44.072605 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.073719 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.076459 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.077713 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.077868 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:44.077996 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:44.078618 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 18:41:44.078787 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4477,"thread_start_us":5505,"threads_started":1}
I20260927 18:41:44.079326 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:44.064279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:33811 (local address 127.2.95.67:45473)
0927 18:41:44.064927 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:44.064935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:44.071532 (+ 6597us) server_negotiation.cc:407] Connection header received
0927 18:41:44.071720 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:44.071725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:44.071806 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:44.071881 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:44.073153 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.074375 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.076813 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.078154 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.078281 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:44.078388 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:44.078933 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 18:41:44.079082 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461}
I20260927 18:41:44.079344 16637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a2a32871c7486ca3cd50727a3a7300" candidate_uuid: "f7dd6df161e044cf9ef045aab61a342d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8576c925fa0341efb839f8d5be77f89b"
I20260927 18:41:44.079636 16637 raft_consensus.cc:3037] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:44.080035 16846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:44.063805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:43829 (local address 127.2.95.65:40271)
0927 18:41:44.070789 (+ 6984us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:44.070808 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:44.070982 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:44.071866 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:44.071871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:44.071896 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:44.072439 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:44.072452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.073878 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.073883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:44.076265 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:44.076281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.078180 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.078188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:44.078401 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:44.078438 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:41:44.079353 (+ 915us) client_negotiation.cc:240] Negotiation successful
0927 18:41:44.079532 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6855,"mutex_wait_us":25,"thread_start_us":7458,"threads_started":1}
I20260927 18:41:44.080142 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:44.063805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:45473 (local address 127.2.95.65:33811)
0927 18:41:44.071427 (+ 7622us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:44.071442 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:44.071567 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:44.072085 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:44.072090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:44.072113 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:44.072965 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:44.072981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.074533 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.074540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:44.076633 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:44.076649 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.078553 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.078559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:44.078723 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:44.078751 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 18:41:44.079495 (+ 744us) client_negotiation.cc:240] Negotiation successful
0927 18:41:44.079673 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7487,"mutex_wait_us":392,"thread_start_us":6310,"threads_started":1}
I20260927 18:41:44.080261 16841 leader_election.cc:288] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [CANDIDATE]: Term 1 election: Requested vote from peers e5ae41da21564291b49e59721491b493 (127.2.95.65:41223), 8576c925fa0341efb839f8d5be77f89b (127.2.95.66:43829)
I20260927 18:41:44.080363 16503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a2a32871c7486ca3cd50727a3a7300" candidate_uuid: "f7dd6df161e044cf9ef045aab61a342d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ae41da21564291b49e59721491b493"
I20260927 18:41:44.080606 16503 raft_consensus.cc:3037] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:44.081255 16771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a2a32871c7486ca3cd50727a3a7300" candidate_uuid: "e5ae41da21564291b49e59721491b493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7dd6df161e044cf9ef045aab61a342d" is_pre_election: true
I20260927 18:41:44.081781 16771 raft_consensus.cc:2379] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5ae41da21564291b49e59721491b493 in current term 1: Already voted for candidate f7dd6df161e044cf9ef045aab61a342d in this term.
I20260927 18:41:44.082798 16636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a2a32871c7486ca3cd50727a3a7300" candidate_uuid: "e5ae41da21564291b49e59721491b493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8576c925fa0341efb839f8d5be77f89b" is_pre_election: true
I20260927 18:41:44.083873 16439 leader_election.cc:302] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5ae41da21564291b49e59721491b493; no voters: 8576c925fa0341efb839f8d5be77f89b, f7dd6df161e044cf9ef045aab61a342d
I20260927 18:41:44.084211 16503 raft_consensus.cc:2454] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7dd6df161e044cf9ef045aab61a342d in term 1.
I20260927 18:41:44.084210 16637 raft_consensus.cc:2454] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7dd6df161e044cf9ef045aab61a342d in term 1.
I20260927 18:41:44.084683 16707 leader_election.cc:302] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8576c925fa0341efb839f8d5be77f89b, f7dd6df161e044cf9ef045aab61a342d; no voters:
W20260927 18:41:44.084685 16817 tablet.cc:2369] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:41:44.085048 16843 raft_consensus.cc:2726] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 18:41:44.085078 16841 raft_consensus.cc:2781] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:41:44.085726 16841 raft_consensus.cc:686] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 1 LEADER]: Becoming Leader. State: Replica: f7dd6df161e044cf9ef045aab61a342d, State: Running, Role: LEADER
I20260927 18:41:44.086349 16841 consensus_queue.cc:237] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:44.092193 16359 catalog_manager.cc:5495] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d reported cstate change: term changed from 0 to 1, leader changed from <none> to f7dd6df161e044cf9ef045aab61a342d (127.2.95.67). New cstate: current_term: 1 leader_uuid: "f7dd6df161e044cf9ef045aab61a342d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } health_report { overall_health: HEALTHY } } }
I20260927 18:41:44.126139 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:44.119692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.2.95.67:45473)
0927 18:41:44.119909 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:44.119918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:44.119996 (+ 78us) server_negotiation.cc:407] Connection header received
0927 18:41:44.120285 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:44.120348 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:44.120492 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:44.120655 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:41:44.121098 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.121962 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.123592 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.124030 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.124143 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:44.124704 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:41:44.125408 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:41:44.125556 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:44.125661 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 18:41:44.125793 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:41:44.137352 16637 raft_consensus.cc:1261] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b [term 1 FOLLOWER]: Refusing update from remote peer f7dd6df161e044cf9ef045aab61a342d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:41:44.137419 16503 raft_consensus.cc:1261] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [term 1 FOLLOWER]: Refusing update from remote peer f7dd6df161e044cf9ef045aab61a342d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:41:44.138460 16848 consensus_queue.cc:1035] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:41:44.138939 16841 consensus_queue.cc:1035] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:41:44.184546 16358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:41:44.184980 16358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:41:44.190065 16358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Sending DeleteTablet for 3 replicas of tablet f4a2a32871c7486ca3cd50727a3a7300
I20260927 18:41:44.191983 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 16420
I20260927 18:41:44.191897 16751 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a2a32871c7486ca3cd50727a3a7300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:41:44 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:52018
I20260927 18:41:44.192411 16617 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a2a32871c7486ca3cd50727a3a7300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:41:44 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:33012
I20260927 18:41:44.192857 16483 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a2a32871c7486ca3cd50727a3a7300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:41:44 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:38954
I20260927 18:41:44.207414 16861 tablet_replica.cc:330] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d: stopping tablet replica
I20260927 18:41:44.208581 16861 raft_consensus.cc:2229] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:41:44.209937 16861 raft_consensus.cc:2258] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:41:44.213738 16861 ts_tablet_manager.cc:1905] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:41:44.220635 16863 tablet_replica.cc:330] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493: stopping tablet replica
I20260927 18:41:44.221415 16863 raft_consensus.cc:2229] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:41:44.223170 16861 ts_tablet_manager.cc:1918] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:41:44.223465 16861 log.cc:1196] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/f4a2a32871c7486ca3cd50727a3a7300
I20260927 18:41:44.223506 16863 raft_consensus.cc:2258] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:41:44.224277 16861 ts_tablet_manager.cc:1939] T f4a2a32871c7486ca3cd50727a3a7300 P f7dd6df161e044cf9ef045aab61a342d: Deleting consensus metadata
I20260927 18:41:44.227319 16343 catalog_manager.cc:4841] TS f7dd6df161e044cf9ef045aab61a342d (127.2.95.67:45473): tablet f4a2a32871c7486ca3cd50727a3a7300 (table test-table [id=196678b203e34bde8474b0ae66293e33]) successfully deleted
I20260927 18:41:44.228581 16862 tablet_replica.cc:330] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b: stopping tablet replica
I20260927 18:41:44.229218 16862 raft_consensus.cc:2229] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:41:44.229593 16863 ts_tablet_manager.cc:1905] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:41:44.229984 16862 raft_consensus.cc:2258] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:41:44.232494 16862 ts_tablet_manager.cc:1905] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 18:41:44 dist-test-slave-9jll krb5kdc[16314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:41:44.238098 16863 ts_tablet_manager.cc:1918] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:41:44.238498 16863 log.cc:1196] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/f4a2a32871c7486ca3cd50727a3a7300
I20260927 18:41:44.239198 16863 ts_tablet_manager.cc:1939] T f4a2a32871c7486ca3cd50727a3a7300 P e5ae41da21564291b49e59721491b493: Deleting consensus metadata
I20260927 18:41:44.240648 16862 ts_tablet_manager.cc:1918] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:41:44.240921 16862 log.cc:1196] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/f4a2a32871c7486ca3cd50727a3a7300
I20260927 18:41:44.241835 16862 ts_tablet_manager.cc:1939] T f4a2a32871c7486ca3cd50727a3a7300 P 8576c925fa0341efb839f8d5be77f89b: Deleting consensus metadata
I20260927 18:41:44.241884 16344 catalog_manager.cc:4841] TS e5ae41da21564291b49e59721491b493 (127.2.95.65:41223): tablet f4a2a32871c7486ca3cd50727a3a7300 (table test-table [id=196678b203e34bde8474b0ae66293e33]) successfully deleted
I20260927 18:41:44.244033 16346 catalog_manager.cc:4841] TS 8576c925fa0341efb839f8d5be77f89b (127.2.95.66:43829): tablet f4a2a32871c7486ca3cd50727a3a7300 (table test-table [id=196678b203e34bde8474b0ae66293e33]) successfully deleted
I20260927 18:41:44.245476 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:44.194367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.2.95.65:41223)
0927 18:41:44.194604 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:44.194613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:44.217813 (+ 23200us) server_negotiation.cc:407] Connection header received
0927 18:41:44.218490 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:44.218497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:44.218668 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:44.218764 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:44.224127 (+ 5363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.225315 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.228387 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.229145 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.229267 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:44.236186 (+ 6919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:44.236214 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:44.236240 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:44.236306 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:44.241174 (+ 4868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:44.242784 (+ 1610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:44.242796 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:44.242800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:44.243002 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:44.243581 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:44.243586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:44.243589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:44.244283 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:44.244567 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:44.244960 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 18:41:44.245166 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:41:44.246559 16543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:44.394942 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 16420
I20260927 18:41:44.414721 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 16554
Sep 27 18:41:44 dist-test-slave-9jll krb5kdc[16314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:41:44.435873 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:44.415884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.2.95.66:43829)
0927 18:41:44.416132 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:44.416139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:44.416166 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:41:44.416385 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:44.416400 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:44.416545 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:44.416683 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:44.418639 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.419937 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.421518 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.422267 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.422431 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:44.428107 (+ 5676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:44.428131 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:44.428153 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:44.428216 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:44.431730 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:44.432993 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:44.433009 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:44.433015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:44.433203 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:44.433637 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:44.433640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:44.433642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:44.434478 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:44.434797 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:44.435198 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 18:41:44.435486 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:41:44.436856 16677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:44.544931 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 16554
I20260927 18:41:44.570776 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 16688
Sep 27 18:41:44 dist-test-slave-9jll krb5kdc[16314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:41:44.594448 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:44.571757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.2.95.67:45473)
0927 18:41:44.571961 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:44.571967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:44.572019 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:41:44.572243 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:44.572248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:44.572321 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:44.572411 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:44.574576 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.575480 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.577586 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:44.578270 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:44.578392 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:44.584882 (+ 6490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:44.584902 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:44.584931 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:44.585010 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:44.590356 (+ 5346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:44.591632 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:44.591645 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:44.591648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:44.591936 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:44.592475 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:44.592479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:44.592481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:44.593061 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:44.593386 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:44.593860 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 18:41:44.594109 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:41:44.595487 16811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:44.737601 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 16688
I20260927 18:41:44.757372 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 16327
I20260927 18:41:44.758942 16389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:44.887516 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 16327
I20260927 18:41:44.911494 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:39751
--webserver_interface=127.2.95.126
--webserver_port=40849
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:39751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:41:45.334435 16869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:45.334880 16869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:45.335033 16869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:45.343756 16869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:41:45.343926 16869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:45.344014 16869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:45.344096 16869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:41:45.344163 16869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:41:45.356774 16869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:39751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:39751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=40849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:45.358712 16869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:45.360848 16869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:45.391197 16878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:45.391211 16876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:45.391582 16875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:45.392349 16869 server_base.cc:1034] running on GCE node
I20260927 18:41:45.393601 16869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:45.395820 16869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:45.397083 16869 hybrid_clock.cc:648] HybridClock initialized: now 1790534505397009 us; error 66 us; skew 500 ppm
I20260927 18:41:45.397580 16869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:45 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534505, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:45.402009 16869 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:41:45.404237 16869 webserver.cc:460] Webserver started at http://127.2.95.126:40849/ using document root <none> and password file <none>
I20260927 18:41:45.405221 16869 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:45.405371 16869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:45.413362 16869 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:41:45.417454 16885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:45.419694 16869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:41:45.419994 16869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "1ba3f602018f4d85a4f645d2d3df0b95"
format_stamp: "Formatted at 2026-09-27 18:41:41 on dist-test-slave-9jll"
server_key: "cbbb43f18cf600fd56df06b582a39264"
server_key_iv: "d13b41c0e3f313b18fa9335a5ea3064e"
server_key_version: "encryptionkey@0"
I20260927 18:41:45.421459 16869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:45.455483 16869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:45.461323 16869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:45.461704 16869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:45.485745 16869 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:39751
I20260927 18:41:45.485770 16936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:39751 every 8 connection(s)
I20260927 18:41:45.487759 16869 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:41:45.495925 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 16869
I20260927 18:41:45.496946 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:41223
--local_ip_for_outbound_sockets=127.2.95.65
--tserver_master_addrs=127.2.95.126:39751
--webserver_port=41719
--webserver_interface=127.2.95.65
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:41:45.497121 16937 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:41:45.501528 16937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Bootstrap starting.
I20260927 18:41:45.526993 16937 log.cc:824] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:45.542098 16937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:41:45.542722 16937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Bootstrap complete.
I20260927 18:41:45.553550 16937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } }
I20260927 18:41:45.554678 16937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba3f602018f4d85a4f645d2d3df0b95, State: Initialized, Role: FOLLOWER
I20260927 18:41:45.555518 16937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } }
I20260927 18:41:45.555804 16937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:41:45.555944 16937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:41:45.556125 16937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:41:45.560973 16937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } }
I20260927 18:41:45.561719 16937 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba3f602018f4d85a4f645d2d3df0b95; no voters:
I20260927 18:41:45.562491 16937 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:41:45.562692 16941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:41:45.563937 16941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 2 LEADER]: Becoming Leader. State: Replica: 1ba3f602018f4d85a4f645d2d3df0b95, State: Running, Role: LEADER
I20260927 18:41:45.564627 16941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } }
I20260927 18:41:45.565227 16937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:41:45.567519 16943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba3f602018f4d85a4f645d2d3df0b95. Latest consensus state: current_term: 2 leader_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } } }
I20260927 18:41:45.568019 16943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:45.568864 16942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } } }
I20260927 18:41:45.569206 16942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:45.574790 16947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:41:45.586184 16947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:41:45.588902 16947 catalog_manager.cc:1259] Loaded cluster ID: 2524eb8f2d234e679199f2177dbef071
I20260927 18:41:45.589190 16947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:41:45.593562 16947 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 18:41:45.611907 16947 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 18:41:45.623769 16947 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:41:45.626013 16947 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Loaded TSK: 0
I20260927 18:41:45.627957 16947 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:41:45.942909 16939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:45.943338 16939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:45.943495 16939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:45.952420 16939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:45.952622 16939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:45.952819 16939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:41:45.964597 16939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:41223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=41719
--tserver_master_addrs=127.2.95.126:39751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:45.966816 16939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:45.968891 16939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:45.987569 16967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:45.987604 16964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:45.987406 16965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:45.988353 16939 server_base.cc:1034] running on GCE node
I20260927 18:41:45.989183 16939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:45.990578 16939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:45.991827 16939 hybrid_clock.cc:648] HybridClock initialized: now 1790534505991751 us; error 73 us; skew 500 ppm
I20260927 18:41:45.992249 16939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:45 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534505, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:45.996186 16939 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:41:45.998208 16939 webserver.cc:460] Webserver started at http://127.2.95.65:41719/ using document root <none> and password file <none>
I20260927 18:41:45.999104 16939 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:45.999240 16939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:46.007048 16939 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 18:41:46.011483 16974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:46.013792 16939 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.003s
I20260927 18:41:46.014034 16939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "e5ae41da21564291b49e59721491b493"
format_stamp: "Formatted at 2026-09-27 18:41:42 on dist-test-slave-9jll"
server_key: "b1739478475041a6f55a94e732c04cc4"
server_key_iv: "f04af2d9f579c9f7b2dba8784945a60f"
server_key_version: "encryptionkey@0"
I20260927 18:41:46.015393 16939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:46.037192 16939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:46.042961 16939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:46.043252 16939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:46.044478 16939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:46.046235 16939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:46.046345 16939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:46.046446 16939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:46.046511 16939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:46.084714 16939 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:41223
I20260927 18:41:46.084753 17086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:41223 every 8 connection(s)
I20260927 18:41:46.086746 16939 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:41:46.090607 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 16939
I20260927 18:41:46.091488 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:43829
--local_ip_for_outbound_sockets=127.2.95.66
--tserver_master_addrs=127.2.95.126:39751
--webserver_port=43227
--webserver_interface=127.2.95.66
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:41:46 dist-test-slave-9jll krb5kdc[16314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534505, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:46.120427 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:46.090998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:52211 (local address 127.2.95.126:39751)
0927 18:41:46.091793 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:46.091813 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:46.092674 (+ 861us) server_negotiation.cc:407] Connection header received
0927 18:41:46.095385 (+ 2711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:46.095402 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:46.095880 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:46.096284 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:46.098577 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:46.100550 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:46.102560 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:46.103283 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:46.103503 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:46.108455 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:46.108467 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:46.108489 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:46.108546 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:46.112396 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:46.113964 (+ 1568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:46.113976 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:46.113984 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:46.114600 (+ 616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:46.115382 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:46.115386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:46.115389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:46.116120 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:46.116355 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:46.117450 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0927 18:41:46.118569 (+ 1119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":337,"threads_started":1}
I20260927 18:41:46.120501 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:46.091502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:52211)
0927 18:41:46.092438 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:46.092491 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:46.094653 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:46.096652 (+ 1999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:46.096666 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:46.097466 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:46.098259 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:46.098293 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:46.100793 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:46.100798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:46.102374 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:46.102393 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:46.103580 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:46.103589 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:46.103907 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:46.104908 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:46.104940 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:46.107967 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:46.112753 (+ 4786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:46.112769 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:46.112794 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:46.113731 (+ 937us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:46.114867 (+ 1136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:46.114873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:46.114878 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:46.115178 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:46.116309 (+ 1131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:46.116320 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:46.116775 (+ 455us) client_negotiation.cc:769] Sending connection context
0927 18:41:46.117095 (+ 320us) client_negotiation.cc:240] Negotiation successful
0927 18:41:46.118187 (+ 1092us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":351,"threads_started":1}
I20260927 18:41:46.125185 17087 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39751
I20260927 18:41:46.125793 17087 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:46.126895 17087 heartbeater.cc:511] Master 127.2.95.126:39751 requested a full tablet report, sending...
I20260927 18:41:46.130071 16902 ts_manager.cc:194] Registered new tserver with Master: e5ae41da21564291b49e59721491b493 (127.2.95.65:41223)
I20260927 18:41:46.132011 16902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:52211
W20260927 18:41:46.502467 17091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:46.502975 17091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:46.503157 17091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:46.514729 17091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:46.514920 17091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:46.515062 17091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:41:46.527334 17091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:43829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=43227
--tserver_master_addrs=127.2.95.126:39751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:46.529199 17091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:46.531014 17091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:46.548559 17096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:46.548591 17099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:46.548559 17097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:46.549217 17091 server_base.cc:1034] running on GCE node
I20260927 18:41:46.550107 17091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:46.551596 17091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:46.552875 17091 hybrid_clock.cc:648] HybridClock initialized: now 1790534506552798 us; error 65 us; skew 500 ppm
I20260927 18:41:46.553349 17091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:46 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:46.557456 17091 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:41:46.559474 17091 webserver.cc:460] Webserver started at http://127.2.95.66:43227/ using document root <none> and password file <none>
I20260927 18:41:46.560427 17091 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:46.560658 17091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:46.568166 17091 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 18:41:46.572430 17106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:46.574556 17091 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 18:41:46.574795 17091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "8576c925fa0341efb839f8d5be77f89b"
format_stamp: "Formatted at 2026-09-27 18:41:43 on dist-test-slave-9jll"
server_key: "a93f51df1a4145ab19ec9c2c6f74d2cb"
server_key_iv: "79319f13fe6ebda0c2cef7de5f262e03"
server_key_version: "encryptionkey@0"
I20260927 18:41:46.576063 17091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:46.605846 17091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:46.612030 17091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:46.612416 17091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:46.613834 17091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:46.616066 17091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:46.616220 17091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:46.616364 17091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:46.616447 17091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:46.654598 17091 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:43829
I20260927 18:41:46.654656 17218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:43829 every 8 connection(s)
I20260927 18:41:46.656497 17091 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:41:46.664753 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 17091
I20260927 18:41:46.665567 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:45473
--local_ip_for_outbound_sockets=127.2.95.67
--tserver_master_addrs=127.2.95.126:39751
--webserver_port=40041
--webserver_interface=127.2.95.67
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:41:46 dist-test-slave-9jll krb5kdc[16314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534506, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:46.682315 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:46.660123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:57443 (local address 127.2.95.126:39751)
0927 18:41:46.660684 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:46.660689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:46.661935 (+ 1246us) server_negotiation.cc:407] Connection header received
0927 18:41:46.664223 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:46.664230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:46.664347 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:46.664454 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:46.666807 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:46.667941 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:46.669903 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:46.670377 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:46.670521 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:46.674789 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:46.674801 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:46.674805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:46.674858 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:46.678154 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:46.679208 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:46.679213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:46.679216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:46.679423 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:46.679997 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:46.680001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:46.680003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:46.680404 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:46.680543 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:46.681602 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0927 18:41:46.681925 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":274,"threads_started":1}
I20260927 18:41:46.684057 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:46.660793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:57443)
0927 18:41:46.661741 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:46.661797 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:46.663867 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:46.664789 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:46.664803 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:46.665651 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:46.666546 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:46.666570 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:46.668198 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:46.668204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:46.669718 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:46.669736 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:46.670662 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:46.670670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:46.670953 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:46.671730 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:46.671759 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:46.674455 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:46.678336 (+ 3881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:46.678349 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:46.678374 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:46.679037 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:46.679568 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:46.679573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:46.679577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:46.679874 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:46.680537 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:46.680548 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:46.681133 (+ 585us) client_negotiation.cc:769] Sending connection context
0927 18:41:46.681470 (+ 337us) client_negotiation.cc:240] Negotiation successful
0927 18:41:46.682186 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":353,"threads_started":1}
I20260927 18:41:46.686067 17219 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39751
I20260927 18:41:46.686493 17219 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:46.687717 17219 heartbeater.cc:511] Master 127.2.95.126:39751 requested a full tablet report, sending...
I20260927 18:41:46.689939 16902 ts_manager.cc:194] Registered new tserver with Master: 8576c925fa0341efb839f8d5be77f89b (127.2.95.66:43829)
I20260927 18:41:46.691140 16902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:57443
W20260927 18:41:47.084715 17223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:47.085220 17223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:47.085393 17223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:47.096031 17223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:47.096192 17223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:47.096329 17223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:41:47.107614 17223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:45473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=40041
--tserver_master_addrs=127.2.95.126:39751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:47.109339 17223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:47.111124 17223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:47.127179 17229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:47.127250 17231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:47.127135 17228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:47.131827 17223 server_base.cc:1034] running on GCE node
I20260927 18:41:47.132798 17223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:47.134200 17223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:47.135458 17223 hybrid_clock.cc:648] HybridClock initialized: now 1790534507135402 us; error 47 us; skew 500 ppm
I20260927 18:41:47.135877 17223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:41:47.135955 17087 heartbeater.cc:503] Master 127.2.95.126:39751 was elected leader, sending a full tablet report...
Sep 27 18:41:47 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534507, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:47.140189 17223 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:41:47.142897 17223 webserver.cc:460] Webserver started at http://127.2.95.67:40041/ using document root <none> and password file <none>
I20260927 18:41:47.143879 17223 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:47.144034 17223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:47.151382 17223 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 18:41:47.155354 17238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:47.157307 17223 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:41:47.157516 17223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "f7dd6df161e044cf9ef045aab61a342d"
format_stamp: "Formatted at 2026-09-27 18:41:43 on dist-test-slave-9jll"
server_key: "fd889fe38f03e2b7c32ad6d9b9245417"
server_key_iv: "75b5e22fd1c0032d6083669a6c9ddbff"
server_key_version: "encryptionkey@0"
I20260927 18:41:47.158806 17223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:47.178850 17223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:47.185088 17223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:47.185437 17223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:47.186707 17223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:47.188764 17223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:47.188871 17223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:47.188990 17223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:47.189064 17223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:47.227802 17223 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:45473
I20260927 18:41:47.227833 17350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:45473 every 8 connection(s)
I20260927 18:41:47.229727 17223 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:41:47.234841 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 17223
Sep 27 18:41:47 dist-test-slave-9jll krb5kdc[16314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534507, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:41:47.253646 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.235935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.2.95.126:39751)
0927 18:41:47.237070 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:47.237075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:47.237090 (+ 15us) server_negotiation.cc:407] Connection header received
0927 18:41:47.237163 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:47.237168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:47.237247 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:47.237353 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:47.239202 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.240116 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.241601 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.242082 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.242200 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.246178 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:47.246184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:47.246190 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:47.246236 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:47.250130 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:47.251295 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:47.251300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:47.251302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:47.251480 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:47.251929 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:47.251934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:47.251937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:47.252376 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:47.252669 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:47.253013 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 18:41:47.253248 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1020,"thread_start_us":876,"threads_started":1}
I20260927 18:41:47.255832 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.232899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:52093 (local address 127.2.95.126:39751)
0927 18:41:47.233526 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:47.233531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:47.234708 (+ 1177us) server_negotiation.cc:407] Connection header received
0927 18:41:47.237373 (+ 2665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:47.237379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:47.237482 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:47.237590 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:47.239604 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.240580 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.242502 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.243027 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.243157 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.248273 (+ 5116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:47.248282 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:47.248287 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:47.248360 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:47.251854 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:47.252990 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:47.252996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:47.252999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:47.253141 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:47.253725 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:47.253730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:47.253733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:47.254083 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:47.254222 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:47.255219 (+ 997us) server_negotiation.cc:299] Negotiation successful
0927 18:41:47.255478 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":332,"threads_started":1}
I20260927 18:41:47.257351 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.233603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:52093)
0927 18:41:47.234525 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:47.234571 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:47.237052 (+ 2481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:47.237956 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:47.237967 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:47.238616 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:41:47.239392 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:47.239412 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.240777 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.240781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:47.242344 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:47.242359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.243299 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.243307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:47.243714 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.244560 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:41:47.244601 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:41:47.247957 (+ 3356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:41:47.252037 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:47.252050 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:47.252077 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:47.252847 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:47.253261 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:41:47.253265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:41:47.253269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:41:47.253582 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:41:47.254222 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:41:47.254234 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:41:47.254811 (+ 577us) client_negotiation.cc:769] Sending connection context
0927 18:41:47.255105 (+ 294us) client_negotiation.cc:240] Negotiation successful
0927 18:41:47.255764 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":354,"threads_started":1}
I20260927 18:41:47.258888 17351 heartbeater.cc:347] Connected to a master server at 127.2.95.126:39751
I20260927 18:41:47.259238 17351 heartbeater.cc:464] Registering TS with master...
I20260927 18:41:47.260190 17351 heartbeater.cc:511] Master 127.2.95.126:39751 requested a full tablet report, sending...
I20260927 18:41:47.262002 16902 ts_manager.cc:194] Registered new tserver with Master: f7dd6df161e044cf9ef045aab61a342d (127.2.95.67:45473)
I20260927 18:41:47.262985 16902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:52093
I20260927 18:41:47.270169 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:41:47.278837 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.271451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.2.95.126:39751)
0927 18:41:47.271646 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:47.271651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:47.272157 (+ 506us) server_negotiation.cc:407] Connection header received
0927 18:41:47.272441 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:47.272446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:47.272656 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:47.272821 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:41:47.273271 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.274198 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.275833 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.276342 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.276507 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.277159 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:41:47.278050 (+ 891us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:41:47.278196 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:47.278410 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 18:41:47.278505 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 18:41:47.280082 16902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:41:47.289120 16902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:41:47.334493 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.305113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:41223 (local address 127.0.0.1:39872)
0927 18:41:47.323049 (+ 17936us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:47.323069 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:47.323299 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:47.325273 (+ 1974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:47.325282 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:47.325318 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:47.326050 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:47.326070 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.328354 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.328359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:47.331165 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:47.331192 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.333400 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.333409 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:47.333619 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.333655 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:41:47.333785 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 18:41:47.333964 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17783,"mutex_wait_us":114,"thread_start_us":18950,"threads_started":1}
I20260927 18:41:47.334640 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.304986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.2.95.66:43829)
0927 18:41:47.324900 (+ 19914us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:47.324911 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:47.324948 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:41:47.325069 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:47.325078 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:47.325351 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:47.326529 (+ 1178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:47.326840 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.328403 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.330779 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.332679 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.333726 (+ 1047us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.333934 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:47.334041 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 18:41:47.334267 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19710,"thread_start_us":19535,"threads_started":1}
I20260927 18:41:47.335115 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.306096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.2.95.65:41223)
0927 18:41:47.318933 (+ 12837us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:47.318950 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:47.324058 (+ 5108us) server_negotiation.cc:407] Connection header received
0927 18:41:47.324212 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:47.324221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:47.324631 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:47.324876 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:47.326318 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.328185 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.331463 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.333040 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.333258 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.333586 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:47.334113 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 18:41:47.334320 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12582,"thread_start_us":12366,"threads_started":1}
I20260927 18:41:47.335282 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.305913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.2.95.67:45473)
0927 18:41:47.323941 (+ 18028us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:47.323953 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:47.324591 (+ 638us) server_negotiation.cc:407] Connection header received
0927 18:41:47.324710 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:47.324718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:47.325103 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:47.325297 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:47.326698 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.327979 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.331741 (+ 3762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.333193 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.333361 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.333624 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:47.334638 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0927 18:41:47.334853 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17827,"thread_start_us":17647,"threads_started":1}
I20260927 18:41:47.335300 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.304579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:43829 (local address 127.0.0.1:38766)
0927 18:41:47.321913 (+ 17334us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:47.321978 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:47.322298 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:47.325612 (+ 3314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:47.325618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:47.325711 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:47.326306 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:47.326326 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.328609 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.328615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:47.330578 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:47.330597 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.332962 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.332970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:47.333227 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.333327 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 18:41:47.333465 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 18:41:47.333742 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14806,"mutex_wait_us":3,"spinlock_wait_cycles":6016,"thread_start_us":16323,"threads_started":1}
I20260927 18:41:47.336916 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.305194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:45473 (local address 127.0.0.1:58686)
0927 18:41:47.324325 (+ 19131us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:47.324343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:47.324468 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:47.325609 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:47.325614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:47.325638 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:47.326499 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:47.326511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.328152 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.328159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:47.331527 (+ 3368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:47.331540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.333972 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.333979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:47.334188 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.334224 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:41:47.334326 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 18:41:47.334485 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19005,"mutex_wait_us":114,"thread_start_us":17534,"threads_started":1}
I20260927 18:41:47.341061 17286 tablet_service.cc:1467] Processing CreateTablet for tablet d60c38edadc34209a9a1b693b575fc70 (DEFAULT_TABLE table=test-table [id=d5dd0d3c2dd24b0db96f317ea108006e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:41:47.340974 17154 tablet_service.cc:1467] Processing CreateTablet for tablet d60c38edadc34209a9a1b693b575fc70 (DEFAULT_TABLE table=test-table [id=d5dd0d3c2dd24b0db96f317ea108006e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:41:47.341261 17022 tablet_service.cc:1467] Processing CreateTablet for tablet d60c38edadc34209a9a1b693b575fc70 (DEFAULT_TABLE table=test-table [id=d5dd0d3c2dd24b0db96f317ea108006e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:41:47.342877 17286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60c38edadc34209a9a1b693b575fc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:47.342908 17154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60c38edadc34209a9a1b693b575fc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:47.343017 17022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d60c38edadc34209a9a1b693b575fc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:41:47.364035 17368 tablet_bootstrap.cc:492] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d: Bootstrap starting.
I20260927 18:41:47.366001 17369 tablet_bootstrap.cc:492] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493: Bootstrap starting.
I20260927 18:41:47.368003 17370 tablet_bootstrap.cc:492] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b: Bootstrap starting.
I20260927 18:41:47.370779 17369 tablet_bootstrap.cc:654] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:47.370779 17368 tablet_bootstrap.cc:654] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:47.371773 17370 tablet_bootstrap.cc:654] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b: Neither blocks nor log segments found. Creating new log.
I20260927 18:41:47.372449 17368 log.cc:824] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:47.372443 17369 log.cc:824] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:47.372903 17370 log.cc:824] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:47.376621 17370 tablet_bootstrap.cc:492] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b: No bootstrap required, opened a new log
I20260927 18:41:47.376672 17368 tablet_bootstrap.cc:492] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d: No bootstrap required, opened a new log
I20260927 18:41:47.376724 17369 tablet_bootstrap.cc:492] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493: No bootstrap required, opened a new log
I20260927 18:41:47.377041 17370 ts_tablet_manager.cc:1397] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.009s
I20260927 18:41:47.377058 17369 ts_tablet_manager.cc:1397] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 18:41:47.377081 17368 ts_tablet_manager.cc:1397] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260927 18:41:47.386215 17370 raft_consensus.cc:348] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:47.386750 17370 raft_consensus.cc:374] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:47.386860 17370 raft_consensus.cc:729] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8576c925fa0341efb839f8d5be77f89b, State: Initialized, Role: FOLLOWER
I20260927 18:41:47.387645 17370 consensus_queue.cc:260] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:47.389084 17219 heartbeater.cc:503] Master 127.2.95.126:39751 was elected leader, sending a full tablet report...
I20260927 18:41:47.388832 17368 raft_consensus.cc:348] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:47.388832 17369 raft_consensus.cc:348] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:47.389340 17369 raft_consensus.cc:374] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:47.389340 17368 raft_consensus.cc:374] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:41:47.389436 17368 raft_consensus.cc:729] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7dd6df161e044cf9ef045aab61a342d, State: Initialized, Role: FOLLOWER
I20260927 18:41:47.389436 17369 raft_consensus.cc:729] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ae41da21564291b49e59721491b493, State: Initialized, Role: FOLLOWER
I20260927 18:41:47.389961 17370 ts_tablet_manager.cc:1428] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 18:41:47.390224 17369 consensus_queue.cc:260] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:47.390216 17368 consensus_queue.cc:260] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:47.392017 17351 heartbeater.cc:503] Master 127.2.95.126:39751 was elected leader, sending a full tablet report...
I20260927 18:41:47.392620 17368 ts_tablet_manager.cc:1428] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 18:41:47.394522 17369 ts_tablet_manager.cc:1428] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260927 18:41:47.409045 17220 tablet.cc:2369] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:41:47.481844 17352 tablet.cc:2369] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:41:47.518416 17374 raft_consensus.cc:482] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:41:47.518833 17374 raft_consensus.cc:504] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:47.521173 17374 leader_election.cc:288] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5ae41da21564291b49e59721491b493 (127.2.95.65:41223), f7dd6df161e044cf9ef045aab61a342d (127.2.95.67:45473)
I20260927 18:41:47.528820 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.521984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:50819 (local address 127.2.95.67:45473)
0927 18:41:47.522327 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:47.522333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:47.522810 (+ 477us) server_negotiation.cc:407] Connection header received
0927 18:41:47.523052 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:47.523057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:47.523127 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:47.523206 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:47.523888 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.524703 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.526866 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.527826 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.527961 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.528084 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:47.528391 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 18:41:47.528533 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260927 18:41:47.528820 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.521845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:40253 (local address 127.2.95.65:41223)
0927 18:41:47.522120 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:47.522129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:47.523021 (+ 892us) server_negotiation.cc:407] Connection header received
0927 18:41:47.523196 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:47.523200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:47.523269 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:47.523345 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:41:47.524069 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.524868 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.526870 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.527778 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.527914 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.528043 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:47.528361 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 18:41:47.528511 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:41:47.529043 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.521532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:45473 (local address 127.2.95.66:50819)
0927 18:41:47.522682 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:47.522705 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:47.522899 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:47.523245 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:47.523250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:47.523291 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:47.523718 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:47.523739 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.524846 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.524852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:47.526708 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:47.526721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.528050 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.528057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:47.528238 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.528261 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 18:41:47.528341 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 18:41:47.528508 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":50,"thread_start_us":718,"threads_started":1}
I20260927 18:41:47.529072 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.521547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:41223 (local address 127.2.95.66:40253)
0927 18:41:47.522900 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:47.522915 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:41:47.523052 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:41:47.523375 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:41:47.523379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:41:47.523431 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:41:47.523883 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:47.523894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.525030 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.525035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:47.526704 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:41:47.526718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.528017 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.528024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:41:47.528195 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.528222 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 18:41:47.528311 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:41:47.528507 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1245,"mutex_wait_us":106,"thread_start_us":928,"threads_started":1}
I20260927 18:41:47.530459 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60c38edadc34209a9a1b693b575fc70" candidate_uuid: "8576c925fa0341efb839f8d5be77f89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ae41da21564291b49e59721491b493" is_pre_election: true
I20260927 18:41:47.530460 17306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60c38edadc34209a9a1b693b575fc70" candidate_uuid: "8576c925fa0341efb839f8d5be77f89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7dd6df161e044cf9ef045aab61a342d" is_pre_election: true
I20260927 18:41:47.531072 17042 raft_consensus.cc:2454] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8576c925fa0341efb839f8d5be77f89b in term 0.
I20260927 18:41:47.531071 17306 raft_consensus.cc:2454] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8576c925fa0341efb839f8d5be77f89b in term 0.
I20260927 18:41:47.531831 17107 leader_election.cc:302] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8576c925fa0341efb839f8d5be77f89b, f7dd6df161e044cf9ef045aab61a342d; no voters:
I20260927 18:41:47.532289 17374 raft_consensus.cc:2781] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:41:47.532459 17374 raft_consensus.cc:482] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:41:47.532567 17374 raft_consensus.cc:3037] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:47.535173 17374 raft_consensus.cc:504] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:47.535872 17374 leader_election.cc:288] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [CANDIDATE]: Term 1 election: Requested vote from peers e5ae41da21564291b49e59721491b493 (127.2.95.65:41223), f7dd6df161e044cf9ef045aab61a342d (127.2.95.67:45473)
I20260927 18:41:47.536569 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60c38edadc34209a9a1b693b575fc70" candidate_uuid: "8576c925fa0341efb839f8d5be77f89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ae41da21564291b49e59721491b493"
I20260927 18:41:47.536571 17306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d60c38edadc34209a9a1b693b575fc70" candidate_uuid: "8576c925fa0341efb839f8d5be77f89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7dd6df161e044cf9ef045aab61a342d"
I20260927 18:41:47.536816 17306 raft_consensus.cc:3037] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:47.536818 17042 raft_consensus.cc:3037] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:41:47.539402 17306 raft_consensus.cc:2454] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8576c925fa0341efb839f8d5be77f89b in term 1.
I20260927 18:41:47.539418 17042 raft_consensus.cc:2454] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8576c925fa0341efb839f8d5be77f89b in term 1.
I20260927 18:41:47.539976 17108 leader_election.cc:302] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8576c925fa0341efb839f8d5be77f89b, e5ae41da21564291b49e59721491b493, f7dd6df161e044cf9ef045aab61a342d; no voters:
I20260927 18:41:47.540454 17374 raft_consensus.cc:2781] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:41:47.541002 17374 raft_consensus.cc:686] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 1 LEADER]: Becoming Leader. State: Replica: 8576c925fa0341efb839f8d5be77f89b, State: Running, Role: LEADER
I20260927 18:41:47.541698 17374 consensus_queue.cc:237] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } } peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } }
I20260927 18:41:47.547873 16901 catalog_manager.cc:5495] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8576c925fa0341efb839f8d5be77f89b (127.2.95.66). New cstate: current_term: 1 leader_uuid: "8576c925fa0341efb839f8d5be77f89b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8576c925fa0341efb839f8d5be77f89b" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 43829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 } health_report { overall_health: UNKNOWN } } }
I20260927 18:41:47.571346 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.564722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.2.95.66:43829)
0927 18:41:47.564991 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:47.564996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:47.565017 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:41:47.565220 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:47.565225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:47.565372 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:47.565525 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:41:47.565984 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.566827 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.568523 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.568988 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.569093 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.569744 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:41:47.570561 (+ 817us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:41:47.570696 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:47.570964 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 18:41:47.571062 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260927 18:41:47.582852 17042 raft_consensus.cc:1261] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493 [term 1 FOLLOWER]: Refusing update from remote peer 8576c925fa0341efb839f8d5be77f89b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:41:47.582854 17306 raft_consensus.cc:1261] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d [term 1 FOLLOWER]: Refusing update from remote peer 8576c925fa0341efb839f8d5be77f89b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:41:47.584244 17379 consensus_queue.cc:1035] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7dd6df161e044cf9ef045aab61a342d" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 45473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:41:47.584947 17374 consensus_queue.cc:1035] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5ae41da21564291b49e59721491b493" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 41223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 18:41:47.590514 17088 tablet.cc:2369] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:41:47.600790 17383 mvcc.cc:205] Tried to move back new op lower bound from 7334029343045722112 to 7334029342895632384. Current Snapshot: MvccSnapshot[applied={T|T < 7334029343045722112}]
I20260927 18:41:47.619099 16901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:41:47.619567 16901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:41:47.624774 16901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Sending DeleteTablet for 3 replicas of tablet d60c38edadc34209a9a1b693b575fc70
I20260927 18:41:47.626585 17154 tablet_service.cc:1514] Processing DeleteTablet for tablet d60c38edadc34209a9a1b693b575fc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:41:47 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:38766
I20260927 18:41:47.626780 17022 tablet_service.cc:1514] Processing DeleteTablet for tablet d60c38edadc34209a9a1b693b575fc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:41:47 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:39872
I20260927 18:41:47.626847 17286 tablet_service.cc:1514] Processing DeleteTablet for tablet d60c38edadc34209a9a1b693b575fc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:41:47 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:58686
I20260927 18:41:47.628089 17393 tablet_replica.cc:330] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493: stopping tablet replica
I20260927 18:41:47.628779 17393 raft_consensus.cc:2229] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:41:47.629145 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 16939
I20260927 18:41:47.629522 17393 raft_consensus.cc:2258] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:41:47.630887 17392 tablet_replica.cc:330] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b: stopping tablet replica
I20260927 18:41:47.631474 17392 raft_consensus.cc:2229] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:41:47.632347 17392 raft_consensus.cc:2258] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:41:47.632903 17394 tablet_replica.cc:330] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d: stopping tablet replica
I20260927 18:41:47.633438 17394 raft_consensus.cc:2229] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:41:47.633693 17393 ts_tablet_manager.cc:1905] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:41:47.634163 17394 raft_consensus.cc:2258] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:41:47.636865 17394 ts_tablet_manager.cc:1905] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:41:47.641733 17393 ts_tablet_manager.cc:1918] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:41:47.642040 17393 log.cc:1196] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/d60c38edadc34209a9a1b693b575fc70
I20260927 18:41:47.642685 17393 ts_tablet_manager.cc:1939] T d60c38edadc34209a9a1b693b575fc70 P e5ae41da21564291b49e59721491b493: Deleting consensus metadata
I20260927 18:41:47.642304 17392 ts_tablet_manager.cc:1905] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:41:47.645176 17394 ts_tablet_manager.cc:1918] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:41:47.645447 17394 log.cc:1196] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/d60c38edadc34209a9a1b693b575fc70
I20260927 18:41:47.645337 16887 catalog_manager.cc:4841] TS e5ae41da21564291b49e59721491b493 (127.2.95.65:41223): tablet d60c38edadc34209a9a1b693b575fc70 (table test-table [id=d5dd0d3c2dd24b0db96f317ea108006e]) successfully deleted
I20260927 18:41:47.646128 17394 ts_tablet_manager.cc:1939] T d60c38edadc34209a9a1b693b575fc70 P f7dd6df161e044cf9ef045aab61a342d: Deleting consensus metadata
I20260927 18:41:47.648443 16886 catalog_manager.cc:4841] TS f7dd6df161e044cf9ef045aab61a342d (127.2.95.67:45473): tablet d60c38edadc34209a9a1b693b575fc70 (table test-table [id=d5dd0d3c2dd24b0db96f317ea108006e]) successfully deleted
I20260927 18:41:47.650259 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.630263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.2.95.65:41223)
0927 18:41:47.630472 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:47.630480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:47.630551 (+ 71us) server_negotiation.cc:407] Connection header received
0927 18:41:47.630785 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:47.630791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:47.630931 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:47.631037 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:47.633142 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.634324 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.636500 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.637175 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.637348 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.641300 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:47.641325 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:47.641351 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:47.641407 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:47.645929 (+ 4522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:47.646913 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:47.646924 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:47.646928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:47.647128 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:47.647550 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:47.647555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:47.647558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:47.648113 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:47.648448 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:47.649385 (+ 937us) server_negotiation.cc:299] Negotiation successful
0927 18:41:47.649816 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:41:47.651226 17082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:47.653652 17392 ts_tablet_manager.cc:1918] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:41:47.654048 17392 log.cc:1196] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/d60c38edadc34209a9a1b693b575fc70
I20260927 18:41:47.654901 17392 ts_tablet_manager.cc:1939] T d60c38edadc34209a9a1b693b575fc70 P 8576c925fa0341efb839f8d5be77f89b: Deleting consensus metadata
I20260927 18:41:47.657388 16889 catalog_manager.cc:4841] TS 8576c925fa0341efb839f8d5be77f89b (127.2.95.66:43829): tablet d60c38edadc34209a9a1b693b575fc70 (table test-table [id=d5dd0d3c2dd24b0db96f317ea108006e]) successfully deleted
I20260927 18:41:47.790985 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 16939
I20260927 18:41:47.817642 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 17091
I20260927 18:41:47.838950 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.818718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.2.95.66:43829)
0927 18:41:47.818939 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:47.818944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:47.819015 (+ 71us) server_negotiation.cc:407] Connection header received
0927 18:41:47.819264 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:47.819270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:47.819412 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:47.819518 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:47.821861 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.822889 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.825302 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.826053 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.826213 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.830582 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:47.830610 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:47.830637 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:47.830792 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:47.834643 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:47.836101 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:47.836111 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:47.836114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:47.836332 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:47.836924 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:47.836930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:47.836933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:47.837536 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:47.837905 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:47.838299 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 18:41:47.838580 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:41:47.839813 17214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:47.969911 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 17091
I20260927 18:41:47.989421 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 17223
I20260927 18:41:48.007184 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:47.990490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.2.95.67:45473)
0927 18:41:47.990689 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 18:41:47.990694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:41:47.990750 (+ 56us) server_negotiation.cc:407] Connection header received
0927 18:41:47.990973 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:41:47.990978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:41:47.991061 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:41:47.991166 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:41:47.992733 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.993618 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.995457 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:41:47.995977 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:41:47.996097 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:41:47.999469 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:41:47.999491 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:41:47.999514 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:41:47.999571 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:41:48.003263 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:48.004381 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:48.004394 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:48.004396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:48.004637 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:41:48.005102 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:41:48.005108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:41:48.005110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:41:48.005719 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:41:48.006082 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0927 18:41:48.006510 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 18:41:48.006753 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 18:41:48.008157 17346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:48.107538 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 17223
I20260927 18:41:48.123929 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 16869
I20260927 18:41:48.125473 16932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:41:48.262533 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 16869
I20260927 18:41:48.287271 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:39751
--webserver_interface=127.2.95.126
--webserver_port=40849
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:39751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:41:48.718891 17399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:48.719311 17399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:48.719414 17399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:48.727564 17399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:41:48.727725 17399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:48.727779 17399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:48.727833 17399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:41:48.727878 17399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:41:48.739430 17399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:39751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:39751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=40849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:48.741266 17399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:48.742964 17399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:48.757076 17404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:48.757051 17405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:48.757091 17407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:48.758143 17399 server_base.cc:1034] running on GCE node
I20260927 18:41:48.759188 17399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:48.761157 17399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:48.762396 17399 hybrid_clock.cc:648] HybridClock initialized: now 1790534508762316 us; error 73 us; skew 500 ppm
I20260927 18:41:48.762861 17399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:48 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534508, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:48.772296 17399 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:41:48.774504 17399 webserver.cc:460] Webserver started at http://127.2.95.126:40849/ using document root <none> and password file <none>
I20260927 18:41:48.775408 17399 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:48.775549 17399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:48.782823 17399 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 18:41:48.788898 17414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:48.791294 17399 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:41:48.791550 17399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "1ba3f602018f4d85a4f645d2d3df0b95"
format_stamp: "Formatted at 2026-09-27 18:41:41 on dist-test-slave-9jll"
server_key: "cbbb43f18cf600fd56df06b582a39264"
server_key_iv: "d13b41c0e3f313b18fa9335a5ea3064e"
server_key_version: "encryptionkey@0"
I20260927 18:41:48.792882 17399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:48.819438 17399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:48.825505 17399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:48.825899 17399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:48.848352 17399 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:39751
I20260927 18:41:48.848368 17465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:39751 every 8 connection(s)
I20260927 18:41:48.850376 17399 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:41:48.859565 17466 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:41:48.860275 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 17399
I20260927 18:41:48.861406 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:41223
--local_ip_for_outbound_sockets=127.2.95.65
--tserver_master_addrs=127.2.95.126:39751
--webserver_port=41719
--webserver_interface=127.2.95.65
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:41:48.865341 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Bootstrap starting.
I20260927 18:41:48.893033 17466 log.cc:824] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Log is configured to *not* fsync() on all Append() calls
I20260927 18:41:48.913434 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:41:48.914156 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95: Bootstrap complete.
I20260927 18:41:48.925148 17466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } }
I20260927 18:41:48.926406 17466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba3f602018f4d85a4f645d2d3df0b95, State: Initialized, Role: FOLLOWER
I20260927 18:41:48.927309 17466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } }
I20260927 18:41:48.927588 17466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:41:48.927731 17466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:41:48.927932 17466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:41:48.932880 17466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } }
I20260927 18:41:48.933563 17466 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba3f602018f4d85a4f645d2d3df0b95; no voters:
I20260927 18:41:48.934401 17466 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 18:41:48.934625 17470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:41:48.935834 17470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [term 3 LEADER]: Becoming Leader. State: Replica: 1ba3f602018f4d85a4f645d2d3df0b95, State: Running, Role: LEADER
I20260927 18:41:48.936502 17470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } }
I20260927 18:41:48.937161 17466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:41:48.940202 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } } }
I20260927 18:41:48.940615 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:48.940088 17472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba3f602018f4d85a4f645d2d3df0b95. Latest consensus state: current_term: 3 leader_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba3f602018f4d85a4f645d2d3df0b95" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 39751 } } }
I20260927 18:41:48.940820 17472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba3f602018f4d85a4f645d2d3df0b95 [sys.catalog]: This master's current role is: LEADER
I20260927 18:41:48.941727 17476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:41:48.962597 17476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:41:48.964859 17476 catalog_manager.cc:1259] Loaded cluster ID: 2524eb8f2d234e679199f2177dbef071
I20260927 18:41:48.965526 17476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 18:41:48.968390 17476 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6c32fbe9ad google::LogMessage::Fail() at ??:0
@ 0x7f6c32fc188a google::LogMessage::SendToLog() at ??:0
@ 0x7f6c32fbe4de google::LogMessage::Flush() at ??:0
@ 0x7f6c32fbfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6c489f1512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6c489ee976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6c48a76610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6c48a76439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6c35c0a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6c35c0d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6c35c0d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6c35be8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6c38ec26db start_thread at ??:0
@ 0x7f6c31e5861f clone at ??:0
W20260927 18:41:49.341192 17468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:49.341622 17468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:49.341816 17468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:49.350986 17468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:49.351195 17468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:49.351379 17468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:41:49.363672 17468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:41223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=41719
--tserver_master_addrs=127.2.95.126:39751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:49.365850 17468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:49.367697 17468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:49.385473 17507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:49.385433 17510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:49.385433 17508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:49.386590 17468 server_base.cc:1034] running on GCE node
I20260927 18:41:49.387532 17468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:49.389134 17468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:49.390494 17468 hybrid_clock.cc:648] HybridClock initialized: now 1790534509390401 us; error 89 us; skew 500 ppm
I20260927 18:41:49.390990 17468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:49 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534509, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:49.395321 17468 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:41:49.397362 17468 webserver.cc:460] Webserver started at http://127.2.95.65:41719/ using document root <none> and password file <none>
I20260927 18:41:49.398392 17468 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:49.398542 17468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:49.406711 17468 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 18:41:49.410743 17517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:49.412925 17468 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:41:49.413177 17468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "e5ae41da21564291b49e59721491b493"
format_stamp: "Formatted at 2026-09-27 18:41:42 on dist-test-slave-9jll"
server_key: "b1739478475041a6f55a94e732c04cc4"
server_key_iv: "f04af2d9f579c9f7b2dba8784945a60f"
server_key_version: "encryptionkey@0"
I20260927 18:41:49.414613 17468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:49.432618 17468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:49.438201 17468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:49.438565 17468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:49.439862 17468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:49.441618 17468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:49.441779 17468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 18:41:49.441903 17468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:49.441988 17468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:49.481763 17468 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:41223
I20260927 18:41:49.481935 17629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:41223 every 8 connection(s)
I20260927 18:41:49.484109 17468 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:41:49.489117 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 17468
I20260927 18:41:49.490104 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:43829
--local_ip_for_outbound_sockets=127.2.95.66
--tserver_master_addrs=127.2.95.126:39751
--webserver_port=43227
--webserver_interface=127.2.95.66
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:41:49.491451 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:49.488490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:60373)
0927 18:41:49.489584 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:49.489932 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":384,"threads_started":1}
W20260927 18:41:49.492476 17630 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:39751 (0 consecutive failures): Network error: Failed to ping master at 127.2.95.126:39751: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
I20260927 18:41:49.493613 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:49.493211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:40075)
0927 18:41:49.493386 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:49.493478 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 18:41:49.494763 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:49.494325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:37373)
0927 18:41:49.494535 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:49.494628 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260927 18:41:49.495102 17630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:41:49.923173 17633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:49.923633 17633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:49.923799 17633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:49.932821 17633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:49.933045 17633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:49.933259 17633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:41:49.946529 17633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:43829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=43227
--tserver_master_addrs=127.2.95.126:39751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:49.948531 17633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:49.950524 17633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:49.968768 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:49.968848 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:49.968876 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:49.969954 17633 server_base.cc:1034] running on GCE node
I20260927 18:41:49.970856 17633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:49.972333 17633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:49.973615 17633 hybrid_clock.cc:648] HybridClock initialized: now 1790534509973532 us; error 73 us; skew 500 ppm
I20260927 18:41:49.974082 17633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:49 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534509, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:49.978982 17633 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:41:49.981715 17633 webserver.cc:460] Webserver started at http://127.2.95.66:43227/ using document root <none> and password file <none>
I20260927 18:41:49.982803 17633 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:49.982964 17633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:49.991194 17633 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.003s
I20260927 18:41:49.995603 17648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:49.997869 17633 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.002s
I20260927 18:41:49.998118 17633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "8576c925fa0341efb839f8d5be77f89b"
format_stamp: "Formatted at 2026-09-27 18:41:43 on dist-test-slave-9jll"
server_key: "a93f51df1a4145ab19ec9c2c6f74d2cb"
server_key_iv: "79319f13fe6ebda0c2cef7de5f262e03"
server_key_version: "encryptionkey@0"
I20260927 18:41:49.999444 17633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:50.057325 17633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:50.063571 17633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:50.063947 17633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:50.065367 17633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:50.067262 17633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:50.067381 17633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:50.067513 17633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:50.067587 17633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:50.104454 17633 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:43829
I20260927 18:41:50.104525 17760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:43829 every 8 connection(s)
I20260927 18:41:50.106370 17633 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:41:50.112058 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:50.109849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:34263)
0927 18:41:50.110709 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:50.111017 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":298,"threads_started":1}
W20260927 18:41:50.112898 17761 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:39751 (0 consecutive failures): Network error: Failed to ping master at 127.2.95.126:39751: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
I20260927 18:41:50.113973 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:50.113517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:44683)
0927 18:41:50.113750 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:50.113839 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 18:41:50.114936 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:50.114537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:60125)
0927 18:41:50.114675 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:50.114756 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 18:41:50.115187 17761 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:41:50.115796 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 17633
I20260927 18:41:50.116472 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:45473
--local_ip_for_outbound_sockets=127.2.95.67
--tserver_master_addrs=127.2.95.126:39751
--webserver_port=40041
--webserver_interface=127.2.95.67
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:41:50.497012 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:50.495931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:44119)
0927 18:41:50.496614 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:50.496739 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":279,"threads_started":1}
W20260927 18:41:50.527700 17764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:41:50.528124 17764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:41:50.528263 17764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:41:50.536499 17764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:41:50.536677 17764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:41:50.536861 17764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:41:50.549161 17764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:45473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=40041
--tserver_master_addrs=127.2.95.126:39751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:41:50.551168 17764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:41:50.552963 17764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:41:50.569319 17770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:50.569319 17771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:41:50.569406 17773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:41:50.575757 17764 server_base.cc:1034] running on GCE node
I20260927 18:41:50.576702 17764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:41:50.578140 17764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:41:50.579401 17764 hybrid_clock.cc:648] HybridClock initialized: now 1790534510579347 us; error 53 us; skew 500 ppm
I20260927 18:41:50.579840 17764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:41:50 dist-test-slave-9jll krb5kdc[16314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534510, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:41:50.583772 17764 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:41:50.585891 17764 webserver.cc:460] Webserver started at http://127.2.95.67:40041/ using document root <none> and password file <none>
I20260927 18:41:50.586819 17764 fs_manager.cc:359] Metadata directory not provided
I20260927 18:41:50.586941 17764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:41:50.594398 17764 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.007s
I20260927 18:41:50.598060 17780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:50.600040 17764 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:41:50.600325 17764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "f7dd6df161e044cf9ef045aab61a342d"
format_stamp: "Formatted at 2026-09-27 18:41:43 on dist-test-slave-9jll"
server_key: "fd889fe38f03e2b7c32ad6d9b9245417"
server_key_iv: "75b5e22fd1c0032d6083669a6c9ddbff"
server_key_version: "encryptionkey@0"
I20260927 18:41:50.601651 17764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:41:50.629547 17764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:41:50.635434 17764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:41:50.635779 17764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:41:50.637094 17764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:41:50.638985 17764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:41:50.639089 17764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:50.639212 17764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:41:50.639297 17764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:41:50.677487 17764 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:45473
I20260927 18:41:50.677575 17892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:45473 every 8 connection(s)
I20260927 18:41:50.679371 17764 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:41:50.685570 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:50.683241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:49085)
0927 18:41:50.684121 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:50.684434 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":338,"threads_started":1}
W20260927 18:41:50.686518 17893 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:39751 (0 consecutive failures): Network error: Failed to ping master at 127.2.95.126:39751: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
I20260927 18:41:50.687577 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:50.687148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:52473)
0927 18:41:50.687360 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:50.687442 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 18:41:50.688566 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 17764
I20260927 18:41:50.688969 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:50.688485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:56835)
0927 18:41:50.688662 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:50.688777 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 18:41:50.689378 17893 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:41:51.117100 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:51.115976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:47583)
0927 18:41:51.116679 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:51.116799 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":334,"threads_started":1}
I20260927 18:41:51.499639 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:51.498565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:59855)
0927 18:41:51.499251 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:51.499423 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":303,"threads_started":1}
I20260927 18:41:51.691447 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:51.690309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:57593)
0927 18:41:51.691004 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:51.691131 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":358,"threads_started":1}
I20260927 18:41:52.119447 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:52.118381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:55399)
0927 18:41:52.119068 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:52.119169 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":299,"threads_started":1}
I20260927 18:41:52.502179 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:52.500980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:57421)
0927 18:41:52.501809 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:52.501937 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":341,"threads_started":1}
I20260927 18:41:52.693647 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:52.692649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:36593)
0927 18:41:52.693304 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:52.693422 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":292,"threads_started":1}
I20260927 18:41:53.121994 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:53.120858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:33647)
0927 18:41:53.121582 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:53.121750 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":357,"threads_started":1}
I20260927 18:41:53.504686 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:53.503635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:33277)
0927 18:41:53.504295 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:53.504411 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":364,"threads_started":1}
I20260927 18:41:53.695861 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:53.694855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:32987)
0927 18:41:53.695464 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:53.695656 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":311,"threads_started":1}
I20260927 18:41:54.124548 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:54.123470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:33205)
0927 18:41:54.124143 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:54.124322 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":344,"threads_started":1}
I20260927 18:41:54.507194 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:54.506017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:56479)
0927 18:41:54.506774 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:54.506911 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":380,"threads_started":1}
I20260927 18:41:54.697988 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:54.697027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:51797)
0927 18:41:54.697630 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:54.697779 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":298,"threads_started":1}
I20260927 18:41:55.127048 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:55.126076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:55689)
0927 18:41:55.126651 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:55.126791 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":287,"threads_started":1}
I20260927 18:41:55.509478 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:55.508259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:39697)
0927 18:41:55.509005 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:55.509167 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":340,"threads_started":1}
I20260927 18:41:55.700441 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:55.699463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:52197)
0927 18:41:55.700131 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:55.700245 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":347,"threads_started":1}
I20260927 18:41:56.129467 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:56.128374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:42321)
0927 18:41:56.129063 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:56.129223 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":292,"threads_started":1}
I20260927 18:41:56.513877 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:56.511421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:35219)
0927 18:41:56.513014 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:56.513376 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1409,"thread_start_us":393,"threads_started":1}
I20260927 18:41:56.703387 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:56.702111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:44551)
0927 18:41:56.703019 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:56.703162 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":524,"threads_started":1}
I20260927 18:41:57.132104 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:57.131019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:58233)
0927 18:41:57.131697 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:57.131813 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":333,"threads_started":1}
I20260927 18:41:57.516536 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:57.515297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:56533)
0927 18:41:57.516147 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:57.516291 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":418,"threads_started":1}
I20260927 18:41:57.706238 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:57.704982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:45379)
0927 18:41:57.705807 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:57.705939 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":358,"threads_started":1}
I20260927 18:41:58.134956 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:58.133860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:55281)
0927 18:41:58.134621 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:58.134751 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":354,"threads_started":1}
I20260927 18:41:58.519182 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:58.518021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:51511)
0927 18:41:58.518692 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:58.518810 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":324,"threads_started":1}
I20260927 18:41:58.708550 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:58.707405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:51289)
0927 18:41:58.708157 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:58.708282 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":430,"threads_started":1}
I20260927 18:41:59.137283 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:59.136251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:57187)
0927 18:41:59.136939 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:59.137045 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":350,"threads_started":1}
I20260927 18:41:59.521549 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:59.520534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:53643)
0927 18:41:59.521236 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:59.521349 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":380,"threads_started":1}
I20260927 18:41:59.710881 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:41:59.709817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:58131)
0927 18:41:59.710524 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 18:41:59.710652 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":328,"threads_started":1}
I20260927 18:42:00.140110 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:00.138972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:49031)
0927 18:42:00.139644 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:00.139770 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":350,"threads_started":1}
I20260927 18:42:00.523697 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:00.522745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:42243)
0927 18:42:00.523350 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:00.523464 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":283,"threads_started":1}
I20260927 18:42:00.713304 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:00.712151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:47921)
0927 18:42:00.712930 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:00.713066 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":350,"threads_started":1}
I20260927 18:42:01.142808 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:01.141573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:60991)
0927 18:42:01.142378 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:01.142527 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":466,"threads_started":1}
I20260927 18:42:01.526374 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:01.525163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:32891)
0927 18:42:01.525979 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:01.526133 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":404,"threads_started":1}
I20260927 18:42:01.715823 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:01.714701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:33041)
0927 18:42:01.715371 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:01.715509 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":329,"threads_started":1}
I20260927 18:42:02.145608 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:02.144399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:43045)
0927 18:42:02.145222 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:02.145377 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":517,"threads_started":1}
I20260927 18:42:02.528987 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:02.527840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:47449)
0927 18:42:02.528574 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:02.528735 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":329,"threads_started":1}
I20260927 18:42:02.718256 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:02.717184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:58125)
0927 18:42:02.717920 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:02.718043 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":315,"threads_started":1}
I20260927 18:42:03.148500 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:03.147346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:58821)
0927 18:42:03.148123 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:03.148262 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":340,"threads_started":1}
I20260927 18:42:03.531348 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:03.530364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:51501)
0927 18:42:03.530966 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:03.531075 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":311,"threads_started":1}
I20260927 18:42:03.720880 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:03.719745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:48397)
0927 18:42:03.720444 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:03.720592 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":337,"threads_started":1}
I20260927 18:42:04.150873 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:04.149705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:59947)
0927 18:42:04.150437 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:04.150620 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":333,"threads_started":1}
I20260927 18:42:04.534022 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:04.532964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:54289)
0927 18:42:04.533617 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:04.533809 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":318,"threads_started":1}
I20260927 18:42:04.723162 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:04.722157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:51101)
0927 18:42:04.722722 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:04.722839 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":268,"threads_started":1}
I20260927 18:42:05.153625 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:05.152564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.66:52281)
0927 18:42:05.153262 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:05.153400 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":356,"threads_started":1}
I20260927 18:42:05.536427 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:05.535382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.65:53635)
0927 18:42:05.536063 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:05.536200 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":319,"threads_started":1}
I20260927 18:42:05.698184 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 17468
I20260927 18:42:05.725952 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:05.699900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.2.95.65:41223)
0927 18:42:05.700685 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:05.700704 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:05.700768 (+ 64us) server_negotiation.cc:407] Connection header received
0927 18:42:05.703500 (+ 2732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:05.703515 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:05.704193 (+ 678us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:05.704609 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:05.706622 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:05.708264 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:05.710001 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:05.710945 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:05.711153 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:05.714894 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:05.714907 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:05.714933 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:05.715000 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:05.719647 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:05.721230 (+ 1583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:05.721242 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:05.721253 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:05.721574 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:05.722369 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:05.722376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:05.722380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:05.723264 (+ 884us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:05.723819 (+ 555us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:05.724625 (+ 806us) server_negotiation.cc:299] Negotiation successful
0927 18:42:05.725467 (+ 842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":350,"threads_started":1}
I20260927 18:42:05.726540 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:05.725031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:39751 (local address 127.2.95.67:37629)
0927 18:42:05.725931 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:05.726223 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":403,"threads_started":1}
I20260927 18:42:05.728987 17625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:42:05.843786 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 17468
I20260927 18:42:05.868256 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 17633
I20260927 18:42:05.894294 17944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:05.870530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.2.95.66:43829)
0927 18:42:05.871493 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:05.871511 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:05.871721 (+ 210us) server_negotiation.cc:407] Connection header received
0927 18:42:05.874644 (+ 2923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:05.874659 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:05.875254 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:05.875870 (+ 616us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:05.878329 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:05.879998 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:05.881616 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:05.882546 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:05.883135 (+ 589us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:05.886318 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:05.886328 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:05.886349 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:05.886403 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:05.890323 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:05.891257 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:05.891266 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:05.891275 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:05.891494 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:05.891930 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:05.891934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:05.891936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:05.892591 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:05.892922 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:05.893423 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 18:42:05.893971 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":534,"threads_started":1}
I20260927 18:42:05.895869 17756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:42:06.005779 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 17633
I20260927 18:42:06.023169 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 17764
I20260927 18:42:06.053442 17946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:06.024886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.2.95.67:45473)
0927 18:42:06.025939 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:06.025956 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:06.026092 (+ 136us) server_negotiation.cc:407] Connection header received
0927 18:42:06.028610 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:06.028627 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:06.029630 (+ 1003us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:06.030347 (+ 717us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:06.032415 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:06.034736 (+ 2321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:06.036696 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:06.038016 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:06.038258 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:06.042458 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:06.042469 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:06.042489 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:06.042656 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:06.046566 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:06.047870 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:06.048182 (+ 312us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:06.048204 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:06.048621 (+ 417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:06.049655 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:06.049661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:06.049713 (+ 52us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:06.050694 (+ 981us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:06.051031 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:06.052426 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0927 18:42:06.053086 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":487,"threads_started":1}
I20260927 18:42:06.055684 17888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:42:06.157567 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 17764
2026-09-27T18:42:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (26280 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:42:06 dist-test-slave-9jll krb5kdc[17950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:42:06 dist-test-slave-9jll krb5kdc[17950](info): set up 2 sockets
Sep 27 18:42:06 dist-test-slave-9jll krb5kdc[17950](info): commencing operation
krb5kdc: starting...
W20260927 18:42:07.286901 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:42:07 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:42:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:42:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:42:07.628787 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:46029
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:46029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:08.107784 17963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:08.108235 17963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:08.108485 17963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:08.118048 17963 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:42:08.118217 17963 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:42:08.118257 17963 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:42:08.118314 17963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:08.118402 17963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:08.118506 17963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:08.118564 17963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:08.118623 17963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:08.130934 17963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:46029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:46029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:42:08.132879 17963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:08.135124 17963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:08.151283 17969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:08.151415 17968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:08.151440 17971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:08.152206 17963 server_base.cc:1034] running on GCE node
I20260927 18:42:08.153285 17963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:08.155275 17963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:08.156615 17963 hybrid_clock.cc:648] HybridClock initialized: now 1790534528156542 us; error 50 us; skew 500 ppm
I20260927 18:42:08.157066 17963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:42:08 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534528, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:08.161324 17963 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:42:08.163393 17963 webserver.cc:460] Webserver started at http://127.2.95.126:36761/ using document root <none> and password file <none>
I20260927 18:42:08.164301 17963 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:08.164419 17963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:08.164798 17963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:08.168658 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "101f14c6069a4d7c9cd66a5a76f0cf79"
format_stamp: "Formatted at 2026-09-27 18:42:08 on dist-test-slave-9jll"
server_key: "7063e14f1678e7ffa27772f86ea196a8"
server_key_iv: "ad747a4d8336dc0362d4e97a92dc06e8"
server_key_version: "encryptionkey@0"
I20260927 18:42:08.169633 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "101f14c6069a4d7c9cd66a5a76f0cf79"
format_stamp: "Formatted at 2026-09-27 18:42:08 on dist-test-slave-9jll"
server_key: "7063e14f1678e7ffa27772f86ea196a8"
server_key_iv: "ad747a4d8336dc0362d4e97a92dc06e8"
server_key_version: "encryptionkey@0"
I20260927 18:42:08.178064 17963 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260927 18:42:08.183126 17978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:08.185433 17963 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260927 18:42:08.185748 17963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "101f14c6069a4d7c9cd66a5a76f0cf79"
format_stamp: "Formatted at 2026-09-27 18:42:08 on dist-test-slave-9jll"
server_key: "7063e14f1678e7ffa27772f86ea196a8"
server_key_iv: "ad747a4d8336dc0362d4e97a92dc06e8"
server_key_version: "encryptionkey@0"
I20260927 18:42:08.186070 17963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:08.215729 17963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:08.221334 17963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:08.221698 17963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:08.244423 17963 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:46029
I20260927 18:42:08.244445 18029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:46029 every 8 connection(s)
I20260927 18:42:08.246222 17963 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:08.247413 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 17963
I20260927 18:42:08.247679 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:42:08.248476 2429 external_mini_cluster.cc:1428] Setting key 5a49cb653c52cdd5885d58d2448bbc82
I20260927 18:42:08.251428 18030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:08.268862 18030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Bootstrap starting.
Sep 27 18:42:08 dist-test-slave-9jll krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:42:08.273643 18030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:08.275417 18030 log.cc:824] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:08.280187 18030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: No bootstrap required, opened a new log
I20260927 18:42:08.282342 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:08.251099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.2.95.126:46029)
0927 18:42:08.254300 (+ 3201us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:08.254319 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:08.254805 (+ 486us) server_negotiation.cc:407] Connection header received
0927 18:42:08.257147 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:08.257187 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:08.257953 (+ 766us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:08.258574 (+ 621us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:08.260268 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:08.262433 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:08.264988 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:08.266358 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:08.266605 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:08.271736 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:08.271750 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:08.271775 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:08.271843 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:08.276276 (+ 4433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:08.277601 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:08.277612 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:08.277623 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:08.277905 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:08.278393 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:08.278399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:08.278402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:08.279255 (+ 853us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:08.279605 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:08.280058 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 18:42:08.280627 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2910,"thread_start_us":2599,"threads_started":1}
I20260927 18:42:08.290268 18030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } }
I20260927 18:42:08.290786 18030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:08.290871 18030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101f14c6069a4d7c9cd66a5a76f0cf79, State: Initialized, Role: FOLLOWER
I20260927 18:42:08.291517 18030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } }
I20260927 18:42:08.291769 18030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:08.291880 18030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:08.292032 18030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:08.294716 18030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } }
I20260927 18:42:08.295344 18030 leader_election.cc:302] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 101f14c6069a4d7c9cd66a5a76f0cf79; no voters:
I20260927 18:42:08.296155 18030 leader_election.cc:288] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:42:08.296253 18035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:08.298034 18035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 1 LEADER]: Becoming Leader. State: Replica: 101f14c6069a4d7c9cd66a5a76f0cf79, State: Running, Role: LEADER
I20260927 18:42:08.298755 18035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } }
I20260927 18:42:08.299347 18030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:08.302950 18037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } } }
I20260927 18:42:08.302940 18036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 101f14c6069a4d7c9cd66a5a76f0cf79. Latest consensus state: current_term: 1 leader_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } } }
I20260927 18:42:08.303567 18037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:08.303633 18036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:08.306649 18042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:08.314863 18042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:08.328964 18042 catalog_manager.cc:1347] Generated new cluster ID: 5540ac4bc99141ddae91ae3caf24cdeb
I20260927 18:42:08.329255 18042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:08.344830 18042 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:42:08.346602 18042 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:08.356432 18042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Generated new TSK 0
I20260927 18:42:08.357782 18042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:42:08.436878 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:46029
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:08.865536 18056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:08.866087 18056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:08.866248 18056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:08.875856 18056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:08.876060 18056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:08.876240 18056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:42:08.887902 18056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:46029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:42:08.889802 18056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:08.891624 18056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:08.908440 18064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:08.908481 18062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:08.908568 18061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:08.910223 18056 server_base.cc:1034] running on GCE node
I20260927 18:42:08.911168 18056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:08.912617 18056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:08.913932 18056 hybrid_clock.cc:648] HybridClock initialized: now 1790534528913872 us; error 48 us; skew 500 ppm
I20260927 18:42:08.914372 18056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:42:08 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534528, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:08.927242 18056 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:42:08.929417 18056 webserver.cc:460] Webserver started at http://127.2.95.65:38547/ using document root <none> and password file <none>
I20260927 18:42:08.930543 18056 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:08.930711 18056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:08.931114 18056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:08.935226 18056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "235cf9bfacdd4c2fbdd25e22b4878177"
format_stamp: "Formatted at 2026-09-27 18:42:08 on dist-test-slave-9jll"
server_key: "b44fc9f2534c3ae62c1ef2c35de610a6"
server_key_iv: "130e8e90fd3699c96fbacfef5560595a"
server_key_version: "encryptionkey@0"
I20260927 18:42:08.936218 18056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "235cf9bfacdd4c2fbdd25e22b4878177"
format_stamp: "Formatted at 2026-09-27 18:42:08 on dist-test-slave-9jll"
server_key: "b44fc9f2534c3ae62c1ef2c35de610a6"
server_key_iv: "130e8e90fd3699c96fbacfef5560595a"
server_key_version: "encryptionkey@0"
I20260927 18:42:08.943284 18056 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260927 18:42:08.948100 18071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:08.950210 18056 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:42:08.950416 18056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "235cf9bfacdd4c2fbdd25e22b4878177"
format_stamp: "Formatted at 2026-09-27 18:42:08 on dist-test-slave-9jll"
server_key: "b44fc9f2534c3ae62c1ef2c35de610a6"
server_key_iv: "130e8e90fd3699c96fbacfef5560595a"
server_key_version: "encryptionkey@0"
I20260927 18:42:08.950706 18056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:08.972178 18056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:08.978682 18056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:08.979054 18056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:08.980396 18056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:08.982225 18056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:08.982314 18056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:08.982515 18056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:08.982625 18056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:09.020936 18056 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:39273
I20260927 18:42:09.021054 18183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:39273 every 8 connection(s)
I20260927 18:42:09.022780 18056 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:09.026305 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 18056
I20260927 18:42:09.026554 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:42:09.027396 2429 external_mini_cluster.cc:1428] Setting key 9e65e3d8796610cc0634d8e977cc3a8c
Sep 27 18:42:09 dist-test-slave-9jll krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534528, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:42:09.053395 18186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:09.026506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:35493 (local address 127.2.95.126:46029)
0927 18:42:09.027121 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:09.027128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:09.028414 (+ 1286us) server_negotiation.cc:407] Connection header received
0927 18:42:09.031057 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:09.031062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:09.031150 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:09.031262 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:09.034220 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:09.035138 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:09.037823 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:09.038323 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:09.038464 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:09.044362 (+ 5898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:09.044369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:09.044373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:09.044434 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:09.048130 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:09.049720 (+ 1590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:09.049727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:09.049731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:09.050035 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:09.050747 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:09.050752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:09.050754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:09.051201 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:09.051341 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:09.052683 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0927 18:42:09.053041 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":280,"threads_started":1}
I20260927 18:42:09.055120 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:09.027220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:35493)
0927 18:42:09.028213 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:09.028260 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:09.030652 (+ 2392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:09.031617 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:09.031634 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:09.032637 (+ 1003us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:09.033919 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:09.033955 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:09.035352 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:09.035357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:09.037536 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:09.037560 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:09.038626 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:09.038633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:09.039001 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:09.040007 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:09.040042 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:09.043897 (+ 3855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:09.048506 (+ 4609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:09.048523 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:09.048554 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:09.049446 (+ 892us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:09.050181 (+ 735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:09.050187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:09.050191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:09.050595 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:09.051384 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:09.051395 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:09.052060 (+ 665us) client_negotiation.cc:769] Sending connection context
0927 18:42:09.052480 (+ 420us) client_negotiation.cc:240] Negotiation successful
0927 18:42:09.053243 (+ 763us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":373,"threads_started":1}
I20260927 18:42:09.057788 18184 heartbeater.cc:347] Connected to a master server at 127.2.95.126:46029
I20260927 18:42:09.058229 18184 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:09.059374 18184 heartbeater.cc:511] Master 127.2.95.126:46029 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
I20260927 18:42:09.062659 17995 ts_manager.cc:194] Registered new tserver with Master: 235cf9bfacdd4c2fbdd25e22b4878177 (127.2.95.65:39273)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
I20260927 18:42:09.064631 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:35493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:42:09.102378 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:46029
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:09.524425 18190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:09.524847 18190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:09.525019 18190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:09.533222 18190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:09.533376 18190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:09.533653 18190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:42:09.546079 18190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:46029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:42:09.547865 18190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:09.549522 18190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:09.567142 18196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:09.567224 18198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:09.567196 18195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:09.568178 18190 server_base.cc:1034] running on GCE node
I20260927 18:42:09.569024 18190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:09.570590 18190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:09.572218 18190 hybrid_clock.cc:648] HybridClock initialized: now 1790534529572112 us; error 101 us; skew 500 ppm
I20260927 18:42:09.572705 18190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:42:09 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534529, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:09.578631 18190 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:42:09.583765 18190 webserver.cc:460] Webserver started at http://127.2.95.66:37607/ using document root <none> and password file <none>
I20260927 18:42:09.584822 18190 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:09.584965 18190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:09.585378 18190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:09.589370 18190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "3072bde1acd14db19da1948d5b55c599"
format_stamp: "Formatted at 2026-09-27 18:42:09 on dist-test-slave-9jll"
server_key: "ca9d3932df951d780365933fc4de9c00"
server_key_iv: "04ec95db9a8f1bd716ec40b3d4ab5f99"
server_key_version: "encryptionkey@0"
I20260927 18:42:09.590454 18190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "3072bde1acd14db19da1948d5b55c599"
format_stamp: "Formatted at 2026-09-27 18:42:09 on dist-test-slave-9jll"
server_key: "ca9d3932df951d780365933fc4de9c00"
server_key_iv: "04ec95db9a8f1bd716ec40b3d4ab5f99"
server_key_version: "encryptionkey@0"
I20260927 18:42:09.597813 18190 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260927 18:42:09.602998 18205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:09.605240 18190 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260927 18:42:09.605506 18190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "3072bde1acd14db19da1948d5b55c599"
format_stamp: "Formatted at 2026-09-27 18:42:09 on dist-test-slave-9jll"
server_key: "ca9d3932df951d780365933fc4de9c00"
server_key_iv: "04ec95db9a8f1bd716ec40b3d4ab5f99"
server_key_version: "encryptionkey@0"
I20260927 18:42:09.605872 18190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:09.632112 18190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:09.637748 18190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:09.638099 18190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:09.639392 18190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:09.641155 18190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:09.641244 18190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:09.641388 18190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:09.641446 18190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:09.677765 18190 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:37393
I20260927 18:42:09.677824 18317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:37393 every 8 connection(s)
I20260927 18:42:09.679468 18190 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:09.686918 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 18190
I20260927 18:42:09.687137 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:42:09.687762 2429 external_mini_cluster.cc:1428] Setting key e0b71318f5bf3752294fb915eef4b62a
Sep 27 18:42:09 dist-test-slave-9jll krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534529, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:42:09.707612 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:09.683002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:33933 (local address 127.2.95.126:46029)
0927 18:42:09.683615 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:09.683621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:09.684932 (+ 1311us) server_negotiation.cc:407] Connection header received
0927 18:42:09.687625 (+ 2693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:09.687632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:09.687733 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:09.687867 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:09.690328 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:09.691341 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:09.693359 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:09.694012 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:09.694157 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:09.699686 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:09.699692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:09.699715 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:09.699766 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:09.703305 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:09.704523 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:09.704530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:09.704533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:09.704717 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:09.705256 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:09.705261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:09.705264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:09.705710 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:09.705865 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:09.706887 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0927 18:42:09.707178 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":297,"threads_started":1}
I20260927 18:42:09.708930 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:09.683638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:33933)
0927 18:42:09.684692 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:09.684755 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:09.687170 (+ 2415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:09.688235 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:09.688249 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:09.689034 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:09.690115 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:09.690156 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:09.691568 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:09.691573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:09.693176 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:09.693193 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:09.694350 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:09.694358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:09.694712 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:09.695675 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:09.695710 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:09.699264 (+ 3554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:09.703558 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:09.703573 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:09.703601 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:09.704360 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:09.704876 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:09.704880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:09.704883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:09.705126 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:09.705899 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:09.705909 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:09.706424 (+ 515us) client_negotiation.cc:769] Sending connection context
0927 18:42:09.706727 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 18:42:09.707364 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":366,"threads_started":1}
I20260927 18:42:09.711176 18318 heartbeater.cc:347] Connected to a master server at 127.2.95.126:46029
I20260927 18:42:09.711608 18318 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:09.712627 18318 heartbeater.cc:511] Master 127.2.95.126:46029 requested a full tablet report, sending...
I20260927 18:42:09.714882 17995 ts_manager.cc:194] Registered new tserver with Master: 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393)
I20260927 18:42:09.715942 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:33933
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:42:09.757980 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:46029
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:42:10.069288 18184 heartbeater.cc:503] Master 127.2.95.126:46029 was elected leader, sending a full tablet report...
W20260927 18:42:10.214371 18324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:10.214769 18324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:10.214891 18324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:10.224112 18324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:10.224298 18324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:10.224431 18324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:42:10.236289 18324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:46029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:42:10.238111 18324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:10.239902 18324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:10.258600 18329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:10.258574 18330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:10.258553 18332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:10.259647 18324 server_base.cc:1034] running on GCE node
I20260927 18:42:10.260591 18324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:10.261863 18324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:10.263100 18324 hybrid_clock.cc:648] HybridClock initialized: now 1790534530263041 us; error 56 us; skew 500 ppm
I20260927 18:42:10.263525 18324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:42:10 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:10.267908 18324 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:42:10.270170 18324 webserver.cc:460] Webserver started at http://127.2.95.67:42897/ using document root <none> and password file <none>
I20260927 18:42:10.271173 18324 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:10.271315 18324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:10.271703 18324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:42:10.275708 18324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "9dbd5308b3c147049c91a913afcfb9bb"
format_stamp: "Formatted at 2026-09-27 18:42:10 on dist-test-slave-9jll"
server_key: "ce7a1c458719deed0c758790f63168f4"
server_key_iv: "8e051d2c7caeac7d6eb4e0c8c02cea09"
server_key_version: "encryptionkey@0"
I20260927 18:42:10.276750 18324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "9dbd5308b3c147049c91a913afcfb9bb"
format_stamp: "Formatted at 2026-09-27 18:42:10 on dist-test-slave-9jll"
server_key: "ce7a1c458719deed0c758790f63168f4"
server_key_iv: "8e051d2c7caeac7d6eb4e0c8c02cea09"
server_key_version: "encryptionkey@0"
I20260927 18:42:10.284037 18324 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 18:42:10.289294 18339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:10.291447 18324 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:42:10.291664 18324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "9dbd5308b3c147049c91a913afcfb9bb"
format_stamp: "Formatted at 2026-09-27 18:42:10 on dist-test-slave-9jll"
server_key: "ce7a1c458719deed0c758790f63168f4"
server_key_iv: "8e051d2c7caeac7d6eb4e0c8c02cea09"
server_key_version: "encryptionkey@0"
I20260927 18:42:10.291921 18324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:10.351238 18324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:10.356709 18324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:10.357040 18324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:10.358270 18324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:10.359889 18324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:10.359987 18324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:10.360138 18324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:10.360198 18324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:10.396811 18324 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:36239
I20260927 18:42:10.396858 18451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:36239 every 8 connection(s)
I20260927 18:42:10.398533 18324 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:10.403759 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 18324
I20260927 18:42:10.404032 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:42:10.404692 2429 external_mini_cluster.cc:1428] Setting key e450366fad33f4c7265fadbadc1b42de
Sep 27 18:42:10 dist-test-slave-9jll krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:42:10.425586 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.402123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:41125 (local address 127.2.95.126:46029)
0927 18:42:10.402712 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:10.402717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:10.404209 (+ 1492us) server_negotiation.cc:407] Connection header received
0927 18:42:10.407221 (+ 3012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:10.407228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:10.407367 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:10.407481 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:10.410006 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.411005 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.413009 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.413493 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.413619 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.417774 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:10.417780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:10.417784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:10.417833 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:10.421415 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:10.422629 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:10.422634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:10.422637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:10.422785 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:10.423401 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:10.423405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:10.423408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:10.423764 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:10.423929 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:10.424978 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0927 18:42:10.425214 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":283,"threads_started":1}
I20260927 18:42:10.427032 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.402771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:41125)
0927 18:42:10.403920 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:10.404054 (+ 134us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:10.406767 (+ 2713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:10.407863 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:10.407876 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:10.408757 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:10.409760 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:10.409796 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.411231 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.411238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:10.412825 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:10.412841 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.413773 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.413779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:10.414050 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.414819 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:10.414847 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:10.417402 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:10.421627 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:10.421637 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:10.421659 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:10.422490 (+ 831us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:10.423003 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:10.423007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:10.423010 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:10.423247 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:10.423981 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:10.423990 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:10.424525 (+ 535us) client_negotiation.cc:769] Sending connection context
0927 18:42:10.424852 (+ 327us) client_negotiation.cc:240] Negotiation successful
0927 18:42:10.425432 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":444,"threads_started":1}
I20260927 18:42:10.429106 18452 heartbeater.cc:347] Connected to a master server at 127.2.95.126:46029
I20260927 18:42:10.429471 18452 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:10.430541 18452 heartbeater.cc:511] Master 127.2.95.126:46029 requested a full tablet report, sending...
I20260927 18:42:10.432650 17995 ts_manager.cc:194] Registered new tserver with Master: 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239)
I20260927 18:42:10.434188 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:41125
I20260927 18:42:10.435518 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:42:10.458144 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.441830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.2.95.126:46029)
0927 18:42:10.442059 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:10.442068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:10.442509 (+ 441us) server_negotiation.cc:407] Connection header received
0927 18:42:10.442692 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:10.442698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:10.442789 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:10.442875 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:10.444553 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.445370 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.446940 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.447476 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.447586 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.450499 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:10.450505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:10.450509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:10.450568 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:10.454273 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:10.455412 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:10.455417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:10.455420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:10.455603 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:10.456205 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:10.456210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:10.456213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:10.456705 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:10.457021 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:10.457454 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 18:42:10.457784 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 18:42:10.463749 17995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:10.472062 17995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:10.517642 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.488918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:36239 (local address 127.0.0.1:49040)
0927 18:42:10.504234 (+ 15316us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:10.504290 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:10.504600 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:10.506294 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:10.506302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:10.506341 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:10.506830 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:10.506848 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.509788 (+ 2940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.509796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:10.513373 (+ 3577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:10.513406 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.516331 (+ 2925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.516340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:10.516602 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.516659 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 18:42:10.516844 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 18:42:10.517104 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14896,"mutex_wait_us":30,"thread_start_us":15947,"threads_started":1}
I20260927 18:42:10.517709 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.489309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.2.95.67:36239)
0927 18:42:10.504084 (+ 14775us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:10.504093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:10.504467 (+ 374us) server_negotiation.cc:407] Connection header received
0927 18:42:10.504986 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:10.504999 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:10.505384 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:10.505745 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:10.507142 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.509515 (+ 2373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.513757 (+ 4242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.515813 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.515986 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.516318 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:10.517034 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 18:42:10.517273 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14526,"thread_start_us":14321,"threads_started":1}
I20260927 18:42:10.522018 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.490053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.2.95.65:39273)
0927 18:42:10.502998 (+ 12945us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:10.503018 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:10.510619 (+ 7601us) server_negotiation.cc:407] Connection header received
0927 18:42:10.510846 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:10.510857 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:10.511255 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:10.511592 (+ 337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:10.513892 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.515735 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.518555 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.520235 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.520372 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.520668 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:10.521358 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 18:42:10.521578 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12552,"thread_start_us":12987,"threads_started":1}
I20260927 18:42:10.522912 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.489709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39273 (local address 127.0.0.1:54700)
0927 18:42:10.510408 (+ 20699us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:10.510437 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:10.510647 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:10.513031 (+ 2384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:10.513037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:10.513072 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:10.513613 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:10.513627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.515997 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.516003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:10.518380 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:10.518396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.520834 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.520839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:10.521098 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.521142 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 18:42:10.522281 (+ 1139us) client_negotiation.cc:240] Negotiation successful
0927 18:42:10.522454 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20437,"mutex_wait_us":26,"thread_start_us":21037,"threads_started":1}
I20260927 18:42:10.523109 18387 tablet_service.cc:1467] Processing CreateTablet for tablet d6093569cb7c447fa8d1049be76beebc (DEFAULT_TABLE table=test-table [id=672c704f2a31456698a6b9b02644468b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:10.525065 18387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6093569cb7c447fa8d1049be76beebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:10.526834 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.506273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:37393 (local address 127.0.0.1:41734)
0927 18:42:10.516315 (+ 10042us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:10.516339 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:10.516522 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:10.517733 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:10.517737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:10.517769 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:10.518211 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:10.518223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.520319 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.520325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:10.522761 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:10.522777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.525719 (+ 2942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.525727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:10.526086 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.526124 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 18:42:10.526219 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 18:42:10.526407 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9852,"mutex_wait_us":25,"thread_start_us":4875,"threads_started":1}
I20260927 18:42:10.526878 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.506591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.2.95.66:37393)
0927 18:42:10.515822 (+ 9231us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:10.515833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:10.516488 (+ 655us) server_negotiation.cc:407] Connection header received
0927 18:42:10.516710 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:10.516718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:10.517108 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:10.517368 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:10.518451 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.520119 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.522995 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.525434 (+ 2439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.525590 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.525839 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:10.526335 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 18:42:10.526530 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9013,"thread_start_us":8831,"threads_started":1}
I20260927 18:42:10.527318 18119 tablet_service.cc:1467] Processing CreateTablet for tablet d6093569cb7c447fa8d1049be76beebc (DEFAULT_TABLE table=test-table [id=672c704f2a31456698a6b9b02644468b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:10.528884 18119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6093569cb7c447fa8d1049be76beebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:10.533376 18253 tablet_service.cc:1467] Processing CreateTablet for tablet d6093569cb7c447fa8d1049be76beebc (DEFAULT_TABLE table=test-table [id=672c704f2a31456698a6b9b02644468b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:10.535012 18253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6093569cb7c447fa8d1049be76beebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:10.546501 18471 tablet_bootstrap.cc:492] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177: Bootstrap starting.
I20260927 18:42:10.547307 18472 tablet_bootstrap.cc:492] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb: Bootstrap starting.
I20260927 18:42:10.549985 18473 tablet_bootstrap.cc:492] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599: Bootstrap starting.
I20260927 18:42:10.551812 18472 tablet_bootstrap.cc:654] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:10.552016 18471 tablet_bootstrap.cc:654] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:10.553128 18472 log.cc:824] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:10.553529 18471 log.cc:824] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:10.555063 18473 tablet_bootstrap.cc:654] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:10.556528 18473 log.cc:824] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:10.557279 18472 tablet_bootstrap.cc:492] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb: No bootstrap required, opened a new log
I20260927 18:42:10.557626 18472 ts_tablet_manager.cc:1397] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260927 18:42:10.558166 18471 tablet_bootstrap.cc:492] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177: No bootstrap required, opened a new log
I20260927 18:42:10.558593 18471 ts_tablet_manager.cc:1397] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 18:42:10.560138 18473 tablet_bootstrap.cc:492] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599: No bootstrap required, opened a new log
I20260927 18:42:10.560493 18473 ts_tablet_manager.cc:1397] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 18:42:10.567641 18471 raft_consensus.cc:348] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } } peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } }
I20260927 18:42:10.568184 18471 raft_consensus.cc:374] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:10.568339 18471 raft_consensus.cc:729] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235cf9bfacdd4c2fbdd25e22b4878177, State: Initialized, Role: FOLLOWER
I20260927 18:42:10.569118 18471 consensus_queue.cc:260] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } } peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } }
I20260927 18:42:10.569087 18472 raft_consensus.cc:348] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } } peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } }
I20260927 18:42:10.569645 18472 raft_consensus.cc:374] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:10.569829 18472 raft_consensus.cc:729] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbd5308b3c147049c91a913afcfb9bb, State: Initialized, Role: FOLLOWER
I20260927 18:42:10.571122 18472 consensus_queue.cc:260] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } } peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } }
I20260927 18:42:10.572501 18471 ts_tablet_manager.cc:1428] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260927 18:42:10.572186 18473 raft_consensus.cc:348] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } } peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } }
I20260927 18:42:10.572829 18473 raft_consensus.cc:374] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:10.572997 18473 raft_consensus.cc:729] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3072bde1acd14db19da1948d5b55c599, State: Initialized, Role: FOLLOWER
I20260927 18:42:10.573618 18452 heartbeater.cc:503] Master 127.2.95.126:46029 was elected leader, sending a full tablet report...
I20260927 18:42:10.573925 18473 consensus_queue.cc:260] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } } peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } }
I20260927 18:42:10.575119 18472 ts_tablet_manager.cc:1428] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb: Time spent starting tablet: real 0.017s user 0.007s sys 0.008s
I20260927 18:42:10.576246 18318 heartbeater.cc:503] Master 127.2.95.126:46029 was elected leader, sending a full tablet report...
I20260927 18:42:10.577029 18473 ts_tablet_manager.cc:1428] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260927 18:42:10.650900 18453 tablet.cc:2369] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:42:10.682371 18319 tablet.cc:2369] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:10.717765 18479 raft_consensus.cc:482] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:10.718251 18479 raft_consensus.cc:504] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } } peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } }
I20260927 18:42:10.720515 18479 leader_election.cc:288] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239), 235cf9bfacdd4c2fbdd25e22b4878177 (127.2.95.65:39273)
I20260927 18:42:10.728554 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.721062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:46619 (local address 127.2.95.67:36239)
0927 18:42:10.721303 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:10.721309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:10.722039 (+ 730us) server_negotiation.cc:407] Connection header received
0927 18:42:10.722427 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:10.722432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:10.722521 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:10.722617 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:10.723378 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.724230 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.726450 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.727371 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.727485 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.727591 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:10.728038 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 18:42:10.728209 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:42:10.728662 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.720894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:36239 (local address 127.2.95.66:46619)
0927 18:42:10.721870 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:10.721896 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:10.722147 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:10.722646 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:10.722651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:10.722678 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:10.723173 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:10.723187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.724431 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.724437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:10.726211 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:10.726227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.727614 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.727622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:10.727819 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.727859 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 18:42:10.727958 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 18:42:10.728118 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":43,"spinlock_wait_cycles":1408,"thread_start_us":566,"threads_started":1}
I20260927 18:42:10.728957 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.720897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39273 (local address 127.2.95.66:51389)
0927 18:42:10.722268 (+ 1371us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:10.722284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:10.722409 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:10.722888 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:10.722892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:10.722933 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:10.723377 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:10.723390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.724690 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.724695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:10.726421 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:10.726437 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.728076 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.728082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:10.728268 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.728301 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:42:10.728399 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 18:42:10.728577 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"thread_start_us":916,"threads_started":1}
I20260927 18:42:10.729229 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.721062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:51389 (local address 127.2.95.65:39273)
0927 18:42:10.721303 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:10.721310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:10.722395 (+ 1085us) server_negotiation.cc:407] Connection header received
0927 18:42:10.722555 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:10.722559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:10.722646 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:10.722740 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:10.723579 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.724525 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.726630 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.727715 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.727875 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.727989 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:10.728448 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 18:42:10.728610 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 18:42:10.730149 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6093569cb7c447fa8d1049be76beebc" candidate_uuid: "3072bde1acd14db19da1948d5b55c599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dbd5308b3c147049c91a913afcfb9bb" is_pre_election: true
I20260927 18:42:10.730165 18139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6093569cb7c447fa8d1049be76beebc" candidate_uuid: "3072bde1acd14db19da1948d5b55c599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" is_pre_election: true
I20260927 18:42:10.730809 18139 raft_consensus.cc:2454] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3072bde1acd14db19da1948d5b55c599 in term 0.
I20260927 18:42:10.730814 18407 raft_consensus.cc:2454] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3072bde1acd14db19da1948d5b55c599 in term 0.
I20260927 18:42:10.731623 18209 leader_election.cc:302] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235cf9bfacdd4c2fbdd25e22b4878177, 3072bde1acd14db19da1948d5b55c599; no voters:
I20260927 18:42:10.732137 18479 raft_consensus.cc:2781] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:10.732302 18479 raft_consensus.cc:482] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:10.732414 18479 raft_consensus.cc:3037] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:10.735553 18479 raft_consensus.cc:504] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } } peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } }
I20260927 18:42:10.736357 18479 leader_election.cc:288] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [CANDIDATE]: Term 1 election: Requested vote from peers 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239), 235cf9bfacdd4c2fbdd25e22b4878177 (127.2.95.65:39273)
I20260927 18:42:10.737013 18139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6093569cb7c447fa8d1049be76beebc" candidate_uuid: "3072bde1acd14db19da1948d5b55c599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235cf9bfacdd4c2fbdd25e22b4878177"
I20260927 18:42:10.737044 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6093569cb7c447fa8d1049be76beebc" candidate_uuid: "3072bde1acd14db19da1948d5b55c599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dbd5308b3c147049c91a913afcfb9bb"
I20260927 18:42:10.737272 18139 raft_consensus.cc:3037] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:10.737288 18407 raft_consensus.cc:3037] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:10.740015 18407 raft_consensus.cc:2454] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3072bde1acd14db19da1948d5b55c599 in term 1.
I20260927 18:42:10.740015 18139 raft_consensus.cc:2454] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3072bde1acd14db19da1948d5b55c599 in term 1.
I20260927 18:42:10.740571 18206 leader_election.cc:302] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3072bde1acd14db19da1948d5b55c599, 9dbd5308b3c147049c91a913afcfb9bb; no voters:
I20260927 18:42:10.740985 18479 raft_consensus.cc:2781] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:10.741508 18479 raft_consensus.cc:686] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 1 LEADER]: Becoming Leader. State: Replica: 3072bde1acd14db19da1948d5b55c599, State: Running, Role: LEADER
I20260927 18:42:10.742195 18479 consensus_queue.cc:237] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } } peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } }
I20260927 18:42:10.748256 17995 catalog_manager.cc:5495] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3072bde1acd14db19da1948d5b55c599 (127.2.95.66). New cstate: current_term: 1 leader_uuid: "3072bde1acd14db19da1948d5b55c599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } health_report { overall_health: UNKNOWN } } }
W20260927 18:42:10.776556 18185 tablet.cc:2369] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:10.807960 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.801502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.2.95.66:37393)
0927 18:42:10.801785 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:10.801790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:10.801808 (+ 18us) server_negotiation.cc:407] Connection header received
0927 18:42:10.801927 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:10.801931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:10.802029 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:10.802188 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:10.802638 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.803462 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.805108 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.805785 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.805937 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.806542 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:10.807266 (+ 724us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:10.807431 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:10.807531 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 18:42:10.807638 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 18:42:10.819478 18407 raft_consensus.cc:1261] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb [term 1 FOLLOWER]: Refusing update from remote peer 3072bde1acd14db19da1948d5b55c599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:10.819486 18139 raft_consensus.cc:1261] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Refusing update from remote peer 3072bde1acd14db19da1948d5b55c599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:10.820669 18482 consensus_queue.cc:1035] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:10.821107 18479 consensus_queue.cc:1035] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:10.853710 17995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:10.854105 17995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:10.858987 17995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Sending DeleteTablet for 3 replicas of tablet d6093569cb7c447fa8d1049be76beebc
I20260927 18:42:10.860558 18387 tablet_service.cc:1514] Processing DeleteTablet for tablet d6093569cb7c447fa8d1049be76beebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:10 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:49040
I20260927 18:42:10.860545 18119 tablet_service.cc:1514] Processing DeleteTablet for tablet d6093569cb7c447fa8d1049be76beebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:10 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:54700
I20260927 18:42:10.860862 18253 tablet_service.cc:1514] Processing DeleteTablet for tablet d6093569cb7c447fa8d1049be76beebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:10 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:41734
I20260927 18:42:10.861721 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 18056
I20260927 18:42:10.871462 18496 tablet_replica.cc:330] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb: stopping tablet replica
I20260927 18:42:10.871750 18497 tablet_replica.cc:330] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599: stopping tablet replica
I20260927 18:42:10.872193 18497 raft_consensus.cc:2229] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:10.872184 18496 raft_consensus.cc:2229] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:10.872900 18496 raft_consensus.cc:2258] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:10.872921 18497 raft_consensus.cc:2258] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:10.874081 18495 tablet_replica.cc:330] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177: stopping tablet replica
I20260927 18:42:10.874687 18495 raft_consensus.cc:2229] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:10.875411 18495 raft_consensus.cc:2258] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:10.875789 18497 ts_tablet_manager.cc:1905] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:10.878103 18495 ts_tablet_manager.cc:1905] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:10.878257 18496 ts_tablet_manager.cc:1905] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:10.884706 18497 ts_tablet_manager.cc:1918] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:10.885048 18497 log.cc:1196] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/d6093569cb7c447fa8d1049be76beebc
I20260927 18:42:10.885794 18497 ts_tablet_manager.cc:1939] T d6093569cb7c447fa8d1049be76beebc P 3072bde1acd14db19da1948d5b55c599: Deleting consensus metadata
I20260927 18:42:10.886252 18495 ts_tablet_manager.cc:1918] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:10.886555 18495 log.cc:1196] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/d6093569cb7c447fa8d1049be76beebc
I20260927 18:42:10.887235 18495 ts_tablet_manager.cc:1939] T d6093569cb7c447fa8d1049be76beebc P 235cf9bfacdd4c2fbdd25e22b4878177: Deleting consensus metadata
I20260927 18:42:10.888396 17979 catalog_manager.cc:4841] TS 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393): tablet d6093569cb7c447fa8d1049be76beebc (table test-table [id=672c704f2a31456698a6b9b02644468b]) successfully deleted
I20260927 18:42:10.889092 18496 ts_tablet_manager.cc:1918] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:10.889441 17982 catalog_manager.cc:4841] TS 235cf9bfacdd4c2fbdd25e22b4878177 (127.2.95.65:39273): tablet d6093569cb7c447fa8d1049be76beebc (table test-table [id=672c704f2a31456698a6b9b02644468b]) successfully deleted
I20260927 18:42:10.889832 18496 log.cc:1196] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/d6093569cb7c447fa8d1049be76beebc
I20260927 18:42:10.890393 18496 ts_tablet_manager.cc:1939] T d6093569cb7c447fa8d1049be76beebc P 9dbd5308b3c147049c91a913afcfb9bb: Deleting consensus metadata
I20260927 18:42:10.892607 17979 catalog_manager.cc:4841] TS 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239): tablet d6093569cb7c447fa8d1049be76beebc (table test-table [id=672c704f2a31456698a6b9b02644468b]) successfully deleted
Sep 27 18:42:10 dist-test-slave-9jll krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:42:10.907908 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:10.863503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.2.95.65:39273)
0927 18:42:10.863746 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:10.863754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:10.880997 (+ 17243us) server_negotiation.cc:407] Connection header received
0927 18:42:10.881327 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:10.881333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:10.881443 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:10.881549 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:10.891861 (+ 10312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.893086 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.894720 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:10.895399 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:10.895564 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:10.900628 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:10.900653 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:10.900681 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:10.900785 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:10.904630 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:10.905530 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:10.905539 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:10.905542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:10.905762 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:10.906154 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:10.906157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:10.906160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:10.906610 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:10.906958 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:10.907348 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 18:42:10.907565 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 18:42:10.908766 18179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:42:11.008934 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 18056
I20260927 18:42:11.036427 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 18190
Sep 27 18:42:11 dist-test-slave-9jll krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:42:11.056720 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:11.037391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.2.95.66:37393)
0927 18:42:11.037631 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:11.037638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:11.037660 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:42:11.037879 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:11.037883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:11.037969 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:11.038067 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:11.039712 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:11.040611 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:11.042054 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:11.042529 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:11.042648 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:11.048143 (+ 5495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:11.048165 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:11.048188 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:11.048255 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:11.052596 (+ 4341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:11.054026 (+ 1430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:11.054038 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:11.054040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:11.054246 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:11.054736 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:11.054745 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:11.054748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:11.055300 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:11.055638 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:11.056081 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 18:42:11.056312 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 18:42:11.057655 18313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:42:11.198093 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 18190
I20260927 18:42:11.219321 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 18324
Sep 27 18:42:11 dist-test-slave-9jll krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:42:11.239573 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:11.220370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.2.95.67:36239)
0927 18:42:11.220609 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:11.220616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:11.220641 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:42:11.220840 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:11.220845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:11.220937 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:11.221048 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:11.222838 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:11.223654 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:11.225049 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:11.225483 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:11.225604 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:11.231310 (+ 5706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:11.231334 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:11.231363 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:11.231425 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:11.235456 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:11.236624 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:11.236635 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:11.236638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:11.236856 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:11.237438 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:11.237444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:11.237446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:11.238095 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:11.238403 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:11.238880 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 18:42:11.239206 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 18:42:11.240351 18447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:42:11.344823 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 18324
I20260927 18:42:11.361045 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 17963
I20260927 18:42:11.362648 18025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:42:11.491920 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 17963
I20260927 18:42:31.515424 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:46029
--webserver_interface=127.2.95.126
--webserver_port=36761
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:46029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:31.893062 18503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:31.893395 18503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:31.893488 18503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:31.901113 18503 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:42:31.901242 18503 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:42:31.901283 18503 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:42:31.901327 18503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:31.901392 18503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:31.901474 18503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:31.901530 18503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:31.901578 18503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:31.912348 18503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:46029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:46029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=36761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:42:31.913794 18503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:31.915179 18503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:31.942588 18510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:31.942657 18509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:31.942588 18512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:31.943630 18503 server_base.cc:1034] running on GCE node
I20260927 18:42:31.944768 18503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:31.947005 18503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:31.948309 18503 hybrid_clock.cc:648] HybridClock initialized: now 1790534551948211 us; error 65 us; skew 500 ppm
I20260927 18:42:31.948969 18503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:42:31 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534551, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:31.953511 18503 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:42:31.955770 18503 webserver.cc:460] Webserver started at http://127.2.95.126:36761/ using document root <none> and password file <none>
I20260927 18:42:31.956813 18503 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:31.957037 18503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:31.965265 18503 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 18:42:31.969269 18519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:31.971279 18503 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:42:31.971515 18503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "101f14c6069a4d7c9cd66a5a76f0cf79"
format_stamp: "Formatted at 2026-09-27 18:42:08 on dist-test-slave-9jll"
server_key: "7063e14f1678e7ffa27772f86ea196a8"
server_key_iv: "ad747a4d8336dc0362d4e97a92dc06e8"
server_key_version: "encryptionkey@0"
I20260927 18:42:31.972922 18503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:31.997754 18503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:32.003607 18503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:32.003997 18503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:32.028780 18503 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:46029
I20260927 18:42:32.028800 18570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:46029 every 8 connection(s)
I20260927 18:42:32.030768 18503 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:32.037575 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 18503
I20260927 18:42:32.038673 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:39273
--local_ip_for_outbound_sockets=127.2.95.65
--tserver_master_addrs=127.2.95.126:46029
--webserver_port=38547
--webserver_interface=127.2.95.65
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:42:32.040076 18571 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:42:32.045241 18571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Bootstrap starting.
I20260927 18:42:32.070657 18571 log.cc:824] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:32.086838 18571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:42:32.087514 18571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Bootstrap complete.
I20260927 18:42:32.097992 18571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } }
I20260927 18:42:32.099134 18571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101f14c6069a4d7c9cd66a5a76f0cf79, State: Initialized, Role: FOLLOWER
I20260927 18:42:32.100020 18571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } }
I20260927 18:42:32.100319 18571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:32.100447 18571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:32.100636 18571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 1 FOLLOWER]: Advancing to term 2
I20260927 18:42:32.105742 18571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } }
I20260927 18:42:32.106442 18571 leader_election.cc:302] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 101f14c6069a4d7c9cd66a5a76f0cf79; no voters:
I20260927 18:42:32.107208 18571 leader_election.cc:288] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 18:42:32.107393 18575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 18:42:32.108669 18575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 2 LEADER]: Becoming Leader. State: Replica: 101f14c6069a4d7c9cd66a5a76f0cf79, State: Running, Role: LEADER
I20260927 18:42:32.109340 18575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } }
I20260927 18:42:32.110121 18571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:32.112676 18577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 101f14c6069a4d7c9cd66a5a76f0cf79. Latest consensus state: current_term: 2 leader_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } } }
I20260927 18:42:32.113304 18577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:32.112617 18576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } } }
I20260927 18:42:32.113495 18576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:32.127250 18581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:32.139973 18581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:32.142052 18581 catalog_manager.cc:1259] Loaded cluster ID: 5540ac4bc99141ddae91ae3caf24cdeb
I20260927 18:42:32.142833 18581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:32.148295 18581 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:32.150750 18581 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Loaded TSK: 0
I20260927 18:42:32.163170 18592 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:42:32.182873 18581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Generated new TSK 1
I20260927 18:42:32.183379 18581 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:42:32.501595 18573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:32.502059 18573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:32.502226 18573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:32.510859 18573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:32.511044 18573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:32.511230 18573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:42:32.522980 18573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:39273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=38547
--tserver_master_addrs=127.2.95.126:46029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:42:32.524782 18573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:32.526569 18573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:32.543654 18599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:32.543895 18601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:32.544215 18598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:32.545038 18573 server_base.cc:1034] running on GCE node
I20260927 18:42:32.545989 18573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:32.547603 18573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:32.548870 18573 hybrid_clock.cc:648] HybridClock initialized: now 1790534552548800 us; error 51 us; skew 500 ppm
I20260927 18:42:32.549309 18573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:42:32 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534552, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:32.554036 18573 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:42:32.556242 18573 webserver.cc:460] Webserver started at http://127.2.95.65:38547/ using document root <none> and password file <none>
I20260927 18:42:32.557320 18573 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:32.557499 18573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:32.565905 18573 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 18:42:32.569380 18608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:32.571345 18573 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:42:32.571547 18573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "235cf9bfacdd4c2fbdd25e22b4878177"
format_stamp: "Formatted at 2026-09-27 18:42:08 on dist-test-slave-9jll"
server_key: "b44fc9f2534c3ae62c1ef2c35de610a6"
server_key_iv: "130e8e90fd3699c96fbacfef5560595a"
server_key_version: "encryptionkey@0"
I20260927 18:42:32.572896 18573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:32.601330 18573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:32.606878 18573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:32.607213 18573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:32.608497 18573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:32.610292 18573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:32.610409 18573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:32.610526 18573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:32.610625 18573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:32.644096 18573 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:39273
I20260927 18:42:32.644150 18720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:39273 every 8 connection(s)
I20260927 18:42:32.645875 18573 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:32.653223 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 18573
I20260927 18:42:32.654022 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:37393
--local_ip_for_outbound_sockets=127.2.95.66
--tserver_master_addrs=127.2.95.126:46029
--webserver_port=37607
--webserver_interface=127.2.95.66
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:42:32 dist-test-slave-9jll krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534552, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:42:32.675125 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:32.649343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:57733 (local address 127.2.95.126:46029)
0927 18:42:32.650106 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:32.650128 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:32.650874 (+ 746us) server_negotiation.cc:407] Connection header received
0927 18:42:32.653848 (+ 2974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:32.653866 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:32.654397 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:32.654846 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:32.656800 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.658248 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.660032 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.660764 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.661005 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:32.665657 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:32.665694 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:32.665719 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:32.665771 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:32.669278 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:32.670326 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:32.670335 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:32.670341 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:32.670530 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:32.671046 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:32.671050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:32.671052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:32.671660 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:32.671863 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:32.672884 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0927 18:42:32.673353 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":378,"threads_started":1}
I20260927 18:42:32.675160 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:32.649644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:57733)
0927 18:42:32.650652 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:32.650708 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:32.652965 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:32.655125 (+ 2160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:32.655139 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:32.655870 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:32.656566 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:32.656588 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.658455 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.658460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:32.659854 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:32.659879 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:32.661052 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:32.661060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:32.661352 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:32.662235 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:32.662271 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:32.665316 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:32.669515 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:32.669525 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:32.669542 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:32.670165 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:32.670681 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:32.670685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:32.670688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:32.670913 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:32.671835 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:32.671844 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:32.672253 (+ 409us) client_negotiation.cc:769] Sending connection context
0927 18:42:32.672603 (+ 350us) client_negotiation.cc:240] Negotiation successful
0927 18:42:32.673347 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":391,"threads_started":1}
I20260927 18:42:32.680008 18721 heartbeater.cc:347] Connected to a master server at 127.2.95.126:46029
I20260927 18:42:32.680492 18721 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:32.681455 18721 heartbeater.cc:511] Master 127.2.95.126:46029 requested a full tablet report, sending...
I20260927 18:42:32.684818 18536 ts_manager.cc:194] Registered new tserver with Master: 235cf9bfacdd4c2fbdd25e22b4878177 (127.2.95.65:39273)
I20260927 18:42:32.687314 18536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:57733
W20260927 18:42:33.058072 18725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:33.058486 18725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:33.058599 18725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:33.066843 18725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:33.067005 18725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:33.067145 18725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:42:33.079124 18725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:37393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=37607
--tserver_master_addrs=127.2.95.126:46029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:42:33.080886 18725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:33.082608 18725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:33.098819 18730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:33.098798 18731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:33.098798 18733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:33.099825 18725 server_base.cc:1034] running on GCE node
I20260927 18:42:33.100642 18725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:33.101848 18725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:33.103101 18725 hybrid_clock.cc:648] HybridClock initialized: now 1790534553103022 us; error 60 us; skew 500 ppm
I20260927 18:42:33.103482 18725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:42:33 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:33.107352 18725 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:42:33.109241 18725 webserver.cc:460] Webserver started at http://127.2.95.66:37607/ using document root <none> and password file <none>
I20260927 18:42:33.110212 18725 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:33.110375 18725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:33.117992 18725 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 18:42:33.121768 18740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:33.123692 18725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:42:33.123916 18725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "3072bde1acd14db19da1948d5b55c599"
format_stamp: "Formatted at 2026-09-27 18:42:09 on dist-test-slave-9jll"
server_key: "ca9d3932df951d780365933fc4de9c00"
server_key_iv: "04ec95db9a8f1bd716ec40b3d4ab5f99"
server_key_version: "encryptionkey@0"
I20260927 18:42:33.125159 18725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:33.147249 18725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:33.152340 18725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:33.152647 18725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:33.153964 18725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:33.155676 18725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:33.155791 18725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:33.155915 18725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:33.155987 18725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:33.192245 18725 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:37393
I20260927 18:42:33.192283 18852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:37393 every 8 connection(s)
I20260927 18:42:33.194516 18725 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:33.197198 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 18725
I20260927 18:42:33.198072 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:36239
--local_ip_for_outbound_sockets=127.2.95.67
--tserver_master_addrs=127.2.95.126:46029
--webserver_port=42897
--webserver_interface=127.2.95.67
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 18:42:33 dist-test-slave-9jll krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:42:33.224319 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.197994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:37059 (local address 127.2.95.126:46029)
0927 18:42:33.198673 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:33.198682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:33.200102 (+ 1420us) server_negotiation.cc:407] Connection header received
0927 18:42:33.202986 (+ 2884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:33.202992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:33.203112 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:33.203239 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:33.205962 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.207179 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.210183 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.211037 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.211204 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.216284 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:33.216292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:33.216297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:33.216352 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:33.220025 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:33.221033 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:33.221038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:33.221042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:33.221214 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:33.221866 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:33.221870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:33.221873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:33.222294 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:33.222466 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:33.223323 (+ 857us) server_negotiation.cc:299] Negotiation successful
0927 18:42:33.223780 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":353,"threads_started":1}
I20260927 18:42:33.225406 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.198806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:37059)
0927 18:42:33.199844 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:33.199907 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:33.202517 (+ 2610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:33.203641 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:33.203655 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:33.204560 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:33.205653 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.205721 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.207413 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.207418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.209947 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.209969 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.211375 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.211382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.211850 (+ 468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.212773 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:33.212805 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:33.215889 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:33.220242 (+ 4353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:33.220253 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:33.220274 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:33.220881 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:33.221340 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:33.221345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:33.221349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:33.221739 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:33.222430 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:33.222438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:33.222898 (+ 460us) client_negotiation.cc:769] Sending connection context
0927 18:42:33.223197 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 18:42:33.223750 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":371,"threads_started":1}
I20260927 18:42:33.227437 18853 heartbeater.cc:347] Connected to a master server at 127.2.95.126:46029
I20260927 18:42:33.227885 18853 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:33.229089 18853 heartbeater.cc:511] Master 127.2.95.126:46029 requested a full tablet report, sending...
I20260927 18:42:33.231402 18536 ts_manager.cc:194] Registered new tserver with Master: 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393)
I20260927 18:42:33.232400 18536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:37059
W20260927 18:42:33.610751 18857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:33.611172 18857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:33.611315 18857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:33.620119 18857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:33.620296 18857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:33.620465 18857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:42:33.632463 18857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:36239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=42897
--tserver_master_addrs=127.2.95.126:46029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:42:33.634291 18857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:33.635910 18857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:33.652360 18865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:33.652391 18863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:33.652315 18862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:33.652897 18857 server_base.cc:1034] running on GCE node
I20260927 18:42:33.653790 18857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:33.655006 18857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:33.656250 18857 hybrid_clock.cc:648] HybridClock initialized: now 1790534553656187 us; error 49 us; skew 500 ppm
I20260927 18:42:33.656750 18857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:42:33 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:33.665536 18857 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:42:33.668007 18857 webserver.cc:460] Webserver started at http://127.2.95.67:42897/ using document root <none> and password file <none>
I20260927 18:42:33.669220 18857 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:33.669404 18857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:33.679265 18857 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 18:42:33.684386 18872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:33.686956 18857 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 18:42:33.687250 18857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "9dbd5308b3c147049c91a913afcfb9bb"
format_stamp: "Formatted at 2026-09-27 18:42:10 on dist-test-slave-9jll"
server_key: "ce7a1c458719deed0c758790f63168f4"
server_key_iv: "8e051d2c7caeac7d6eb4e0c8c02cea09"
server_key_version: "encryptionkey@0"
I20260927 18:42:33.688925 18857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:33.692286 18721 heartbeater.cc:503] Master 127.2.95.126:46029 was elected leader, sending a full tablet report...
I20260927 18:42:33.736368 18857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:33.742323 18857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:33.742650 18857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:33.744045 18857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:33.745910 18857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:33.746018 18857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:33.746130 18857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:33.746197 18857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:33.783980 18857 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:36239
I20260927 18:42:33.784143 18984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:36239 every 8 connection(s)
I20260927 18:42:33.785748 18857 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:33.788717 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 18857
Sep 27 18:42:33 dist-test-slave-9jll krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534553, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:42:33.808715 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.789800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.2.95.126:46029)
0927 18:42:33.790318 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:33.790325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:33.790702 (+ 377us) server_negotiation.cc:407] Connection header received
0927 18:42:33.790935 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:33.790941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:33.791063 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:33.791203 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:33.793456 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.794641 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.796635 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.797697 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.797867 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.801792 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:33.801798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:33.801803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:33.801860 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:33.804791 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:33.805987 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:33.805993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:33.805996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:33.806197 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:33.806814 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:33.806818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:33.806821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:33.807194 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:33.807655 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:33.808053 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 18:42:33.808309 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":398,"thread_start_us":258,"threads_started":1}
I20260927 18:42:33.814522 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.789106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:34093 (local address 127.2.95.126:46029)
0927 18:42:33.789776 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:33.789784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:33.791253 (+ 1469us) server_negotiation.cc:407] Connection header received
0927 18:42:33.794153 (+ 2900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:33.794160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:33.794280 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:33.794413 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:33.796991 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.798457 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.801194 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.801753 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.801894 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.806397 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:33.806406 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:33.806410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:33.806457 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:33.810560 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:33.811826 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:33.811831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:33.811833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:33.811971 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:33.812629 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:33.812633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:33.812636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:33.812939 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:33.813067 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:33.813896 (+ 829us) server_negotiation.cc:299] Negotiation successful
0927 18:42:33.814168 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":472,"thread_start_us":295,"threads_started":1}
I20260927 18:42:33.815732 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.789857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:34093)
0927 18:42:33.791093 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:33.791140 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:33.793791 (+ 2651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:33.794659 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:33.794670 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:33.795659 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:33.796641 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.796669 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.798626 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.798630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.800950 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.800976 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.802047 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.802055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.802396 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.803082 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:33.803118 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:33.806020 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:33.810778 (+ 4758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:33.810790 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:33.810813 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:33.811681 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:33.812163 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:33.812170 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:33.812173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:33.812483 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:33.813047 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:33.813054 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:33.813479 (+ 425us) client_negotiation.cc:769] Sending connection context
0927 18:42:33.813817 (+ 338us) client_negotiation.cc:240] Negotiation successful
0927 18:42:33.814360 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":475,"threads_started":1}
I20260927 18:42:33.817616 18985 heartbeater.cc:347] Connected to a master server at 127.2.95.126:46029
I20260927 18:42:33.818097 18985 heartbeater.cc:464] Registering TS with master...
I20260927 18:42:33.819064 18985 heartbeater.cc:511] Master 127.2.95.126:46029 requested a full tablet report, sending...
I20260927 18:42:33.821058 18536 ts_manager.cc:194] Registered new tserver with Master: 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239)
I20260927 18:42:33.822203 18536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:34093
I20260927 18:42:33.825608 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:42:33.834240 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.826854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.2.95.126:46029)
0927 18:42:33.827132 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:33.827138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:33.827683 (+ 545us) server_negotiation.cc:407] Connection header received
0927 18:42:33.827897 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:33.827902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:33.828112 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:33.828265 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:33.828834 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.829636 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.831682 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.832339 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.832460 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.833192 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:33.833491 (+ 299us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 18:42:33.833933 (+ 442us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40770: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 18:42:33.834388 18987 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40770: token expired
W20260927 18:42:33.835117 18456 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.95.126:46029)
I20260927 18:42:33.850656 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.835818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.2.95.126:46029)
0927 18:42:33.836038 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:33.836044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:33.836063 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:42:33.836225 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:33.836229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:33.836304 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:33.836392 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:33.838078 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.838928 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.840527 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.840991 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.841112 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.844014 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:33.844020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:33.844024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:33.844070 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:33.847309 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:33.848312 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:33.848317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:33.848320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:33.848500 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:33.848904 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:33.848907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:33.848909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:33.849314 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:33.849547 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:33.850023 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 18:42:33.850243 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 18:42:33.855535 18536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:42:33.864298 18536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:42:33.892277 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.880097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:37393 (local address 127.0.0.1:56880)
0927 18:42:33.881183 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:33.881226 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:33.881458 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:33.883058 (+ 1600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:33.883067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:33.883100 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:33.883586 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.883604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.885958 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.885964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.888999 (+ 3035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.889022 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.891238 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.891244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.891446 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.891489 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 18:42:33.891636 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 18:42:33.891870 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":1304,"threads_started":1}
I20260927 18:42:33.892300 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.880814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.2.95.66:37393)
0927 18:42:33.882266 (+ 1452us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:33.882277 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:33.882311 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:42:33.882467 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:33.882477 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:33.882800 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:33.883035 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:33.883781 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.885768 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.889276 (+ 3508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.890940 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.891063 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.891350 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:33.891710 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 18:42:33.891903 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1264,"thread_start_us":1025,"threads_started":1}
I20260927 18:42:33.893375 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.882028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:36239 (local address 127.0.0.1:34836)
0927 18:42:33.882648 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:33.882668 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:33.882796 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:33.884968 (+ 2172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:33.884973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:33.885014 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:33.885428 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.885440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.887587 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.887593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.890466 (+ 2873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.890481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.892531 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.892538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.892706 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.892739 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:42:33.892808 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 18:42:33.892957 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":1,"thread_start_us":1166,"threads_started":1}
I20260927 18:42:33.894760 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.882828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34836 (local address 127.2.95.67:36239)
0927 18:42:33.884296 (+ 1468us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:33.884306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:33.884344 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:42:33.884462 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:33.884470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:33.884732 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:33.885693 (+ 961us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:33.886034 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.887358 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.890682 (+ 3324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.892269 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.893800 (+ 1531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.894086 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:33.894180 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 18:42:33.894384 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1273,"thread_start_us":1040,"threads_started":1}
I20260927 18:42:33.894961 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.880183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39273 (local address 127.0.0.1:49868)
0927 18:42:33.881184 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:33.881233 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:33.881429 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:33.886990 (+ 5561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:33.886996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:33.887096 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:33.887731 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.887744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.889732 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.889737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.891713 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.891729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.894205 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.894212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.894388 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.894425 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:42:33.894491 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 18:42:33.894649 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":476,"threads_started":1}
I20260927 18:42:33.896081 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.881445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49868 (local address 127.2.95.65:39273)
0927 18:42:33.886300 (+ 4855us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:33.886310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:33.886345 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:42:33.886470 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:33.886480 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:33.886737 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:33.887010 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:33.887915 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.889569 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.891917 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.893963 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.895211 (+ 1248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.895455 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:33.895545 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 18:42:33.895734 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4624,"thread_start_us":4357,"threads_started":1}
I20260927 18:42:33.897324 18788 tablet_service.cc:1467] Processing CreateTablet for tablet f3a903ef9d8c4457a248f72e9e1e4420 (DEFAULT_TABLE table=test-table [id=eaa78a3f51674f3c8ab3a9a09cc0f7d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:33.899300 18788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a903ef9d8c4457a248f72e9e1e4420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:33.899468 18920 tablet_service.cc:1467] Processing CreateTablet for tablet f3a903ef9d8c4457a248f72e9e1e4420 (DEFAULT_TABLE table=test-table [id=eaa78a3f51674f3c8ab3a9a09cc0f7d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:33.901222 18920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a903ef9d8c4457a248f72e9e1e4420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:33.904793 18656 tablet_service.cc:1467] Processing CreateTablet for tablet f3a903ef9d8c4457a248f72e9e1e4420 (DEFAULT_TABLE table=test-table [id=eaa78a3f51674f3c8ab3a9a09cc0f7d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:42:33.906424 18656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3a903ef9d8c4457a248f72e9e1e4420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:42:33.918283 19002 tablet_bootstrap.cc:492] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599: Bootstrap starting.
I20260927 18:42:33.921336 19003 tablet_bootstrap.cc:492] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb: Bootstrap starting.
I20260927 18:42:33.925463 19003 tablet_bootstrap.cc:654] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:33.925585 19002 tablet_bootstrap.cc:654] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:33.926872 19004 tablet_bootstrap.cc:492] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: Bootstrap starting.
I20260927 18:42:33.927241 19003 log.cc:824] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:33.927649 19002 log.cc:824] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:33.931666 19003 tablet_bootstrap.cc:492] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb: No bootstrap required, opened a new log
I20260927 18:42:33.932029 19003 ts_tablet_manager.cc:1397] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 18:42:33.932224 19002 tablet_bootstrap.cc:492] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599: No bootstrap required, opened a new log
I20260927 18:42:33.932515 19004 tablet_bootstrap.cc:654] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: Neither blocks nor log segments found. Creating new log.
I20260927 18:42:33.932713 19002 ts_tablet_manager.cc:1397] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260927 18:42:33.934216 19004 log.cc:824] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:33.938373 19004 tablet_bootstrap.cc:492] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: No bootstrap required, opened a new log
I20260927 18:42:33.938802 19004 ts_tablet_manager.cc:1397] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260927 18:42:33.941558 19002 raft_consensus.cc:348] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:33.942122 19002 raft_consensus.cc:374] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:33.942238 19002 raft_consensus.cc:729] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3072bde1acd14db19da1948d5b55c599, State: Initialized, Role: FOLLOWER
I20260927 18:42:33.943174 19002 consensus_queue.cc:260] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:33.943117 19003 raft_consensus.cc:348] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:33.943624 19003 raft_consensus.cc:374] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:33.943742 19003 raft_consensus.cc:729] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbd5308b3c147049c91a913afcfb9bb, State: Initialized, Role: FOLLOWER
I20260927 18:42:33.944403 19003 consensus_queue.cc:260] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:33.945509 18853 heartbeater.cc:503] Master 127.2.95.126:46029 was elected leader, sending a full tablet report...
I20260927 18:42:33.946481 19002 ts_tablet_manager.cc:1428] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260927 18:42:33.946846 18854 tablet.cc:2369] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:33.949082 19003 ts_tablet_manager.cc:1428] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260927 18:42:33.950126 18985 heartbeater.cc:503] Master 127.2.95.126:46029 was elected leader, sending a full tablet report...
I20260927 18:42:33.950683 19004 raft_consensus.cc:348] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:33.951243 19004 raft_consensus.cc:374] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:42:33.951363 19004 raft_consensus.cc:729] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235cf9bfacdd4c2fbdd25e22b4878177, State: Initialized, Role: FOLLOWER
I20260927 18:42:33.952183 19004 consensus_queue.cc:260] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:33.954272 19004 ts_tablet_manager.cc:1428] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 18:42:33.970593 19010 raft_consensus.cc:482] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:33.971019 19010 raft_consensus.cc:504] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:33.973227 19010 leader_election.cc:288] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393), 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239)
I20260927 18:42:33.981779 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.973646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:41805 (local address 127.2.95.66:37393)
0927 18:42:33.973932 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:33.973949 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:33.974455 (+ 506us) server_negotiation.cc:407] Connection header received
0927 18:42:33.974889 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:33.974901 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:33.975018 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:33.975123 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:33.975992 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.977087 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.979504 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.980631 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.980741 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.980841 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:33.981262 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 18:42:33.981405 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 18:42:33.981851 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.973522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:37393 (local address 127.2.95.65:41805)
0927 18:42:33.974313 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:33.974335 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:33.974646 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:33.975155 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:33.975159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:33.975185 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:33.975810 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.975826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.977227 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.977233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.979320 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.979334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.980863 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.980869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.981052 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.981089 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 18:42:33.981203 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 18:42:33.981374 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":386,"threads_started":1}
I20260927 18:42:33.982527 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.974549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:48905 (local address 127.2.95.67:36239)
0927 18:42:33.974778 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:33.974785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:33.975968 (+ 1183us) server_negotiation.cc:407] Connection header received
0927 18:42:33.976186 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:33.976190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:33.976260 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:33.976334 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:42:33.976936 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.978054 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.980289 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.981343 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.981458 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.981585 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:33.982140 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 18:42:33.982285 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 18:42:33.983155 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:33.974348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:36239 (local address 127.2.95.65:48905)
0927 18:42:33.975852 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:33.975867 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:33.976008 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:33.976374 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:33.976378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:33.976405 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:42:33.976768 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.976785 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.978204 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.978208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.980108 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:33.980123 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:33.981609 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:33.981616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:33.981829 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:33.981865 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:42:33.982710 (+ 845us) client_negotiation.cc:240] Negotiation successful
0927 18:42:33.982836 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1387,"thread_start_us":1165,"threads_started":1}
I20260927 18:42:33.983085 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a903ef9d8c4457a248f72e9e1e4420" candidate_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3072bde1acd14db19da1948d5b55c599" is_pre_election: true
I20260927 18:42:33.983737 18808 raft_consensus.cc:2454] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 235cf9bfacdd4c2fbdd25e22b4878177 in term 0.
I20260927 18:42:33.984337 18940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a903ef9d8c4457a248f72e9e1e4420" candidate_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dbd5308b3c147049c91a913afcfb9bb" is_pre_election: true
I20260927 18:42:33.984584 18609 leader_election.cc:302] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235cf9bfacdd4c2fbdd25e22b4878177, 3072bde1acd14db19da1948d5b55c599; no voters:
I20260927 18:42:33.984865 18940 raft_consensus.cc:2454] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 235cf9bfacdd4c2fbdd25e22b4878177 in term 0.
I20260927 18:42:33.985145 19010 raft_consensus.cc:2781] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:42:33.985313 19010 raft_consensus.cc:482] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:42:33.985440 19010 raft_consensus.cc:3037] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:33.988086 19010 raft_consensus.cc:504] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:33.988922 19010 leader_election.cc:288] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 1 election: Requested vote from peers 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393), 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239)
I20260927 18:42:33.989739 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a903ef9d8c4457a248f72e9e1e4420" candidate_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3072bde1acd14db19da1948d5b55c599"
I20260927 18:42:33.989845 18940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3a903ef9d8c4457a248f72e9e1e4420" candidate_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dbd5308b3c147049c91a913afcfb9bb"
I20260927 18:42:33.990064 18808 raft_consensus.cc:3037] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:33.990103 18940 raft_consensus.cc:3037] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:42:33.993763 18808 raft_consensus.cc:2454] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235cf9bfacdd4c2fbdd25e22b4878177 in term 1.
I20260927 18:42:33.993763 18940 raft_consensus.cc:2454] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235cf9bfacdd4c2fbdd25e22b4878177 in term 1.
I20260927 18:42:33.994336 18609 leader_election.cc:302] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235cf9bfacdd4c2fbdd25e22b4878177, 3072bde1acd14db19da1948d5b55c599; no voters:
I20260927 18:42:33.994704 19010 raft_consensus.cc:2781] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:42:33.995265 19010 raft_consensus.cc:686] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 LEADER]: Becoming Leader. State: Replica: 235cf9bfacdd4c2fbdd25e22b4878177, State: Running, Role: LEADER
I20260927 18:42:33.995965 19010 consensus_queue.cc:237] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:34.002139 18536 catalog_manager.cc:5495] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 reported cstate change: term changed from 0 to 1, leader changed from <none> to 235cf9bfacdd4c2fbdd25e22b4878177 (127.2.95.65). New cstate: current_term: 1 leader_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } health_report { overall_health: UNKNOWN } } }
I20260927 18:42:34.035996 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.028387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.2.95.65:39273)
0927 18:42:34.028656 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.028662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.028684 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:42:34.028823 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.028829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.028969 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.029167 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:42:34.029581 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.030873 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.032555 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.033238 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.033373 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.034145 (+ 772us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:42:34.035184 (+ 1039us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:42:34.035355 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.035546 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.035665 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 18:42:34.037640 18986 tablet.cc:2369] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:42:34.047664 18940 raft_consensus.cc:1261] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb [term 1 FOLLOWER]: Refusing update from remote peer 235cf9bfacdd4c2fbdd25e22b4878177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:34.047664 18808 raft_consensus.cc:1261] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599 [term 1 FOLLOWER]: Refusing update from remote peer 235cf9bfacdd4c2fbdd25e22b4878177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:42:34.049083 19013 consensus_queue.cc:1035] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:34.049588 19010 consensus_queue.cc:1035] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:42:34.066474 19017 mvcc.cc:205] Tried to move back new op lower bound from 7334029533364379648 to 7334029533172555776. Current Snapshot: MvccSnapshot[applied={T|T < 7334029533364379648}]
I20260927 18:42:34.066789 19018 mvcc.cc:205] Tried to move back new op lower bound from 7334029533364379648 to 7334029533172555776. Current Snapshot: MvccSnapshot[applied={T|T < 7334029533364379648}]
I20260927 18:42:34.086498 18535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:34.086884 18535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:42:34.092537 18535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Sending DeleteTablet for 3 replicas of tablet f3a903ef9d8c4457a248f72e9e1e4420
I20260927 18:42:34.094256 18788 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a903ef9d8c4457a248f72e9e1e4420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:34 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:56880
I20260927 18:42:34.094270 18656 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a903ef9d8c4457a248f72e9e1e4420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:34 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:49868
I20260927 18:42:34.094799 18920 tablet_service.cc:1514] Processing DeleteTablet for tablet f3a903ef9d8c4457a248f72e9e1e4420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:42:34 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:34836
I20260927 18:42:34.095711 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 18573
I20260927 18:42:34.105222 19028 tablet_replica.cc:330] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb: stopping tablet replica
I20260927 18:42:34.105974 19028 raft_consensus.cc:2229] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:34.106724 19028 raft_consensus.cc:2258] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:34.112144 19026 tablet_replica.cc:330] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599: stopping tablet replica
I20260927 18:42:34.112980 19028 ts_tablet_manager.cc:1905] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:34.113003 19026 raft_consensus.cc:2229] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:42:34.113862 19026 raft_consensus.cc:2258] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:34.116757 19026 ts_tablet_manager.cc:1905] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:42:34.119596 19027 tablet_replica.cc:330] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: stopping tablet replica
I20260927 18:42:34.120240 19027 raft_consensus.cc:2229] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:42:34.120819 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.097178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.2.95.65:39273)
0927 18:42:34.097395 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.097401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.097424 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:42:34.097500 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.097506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.097605 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.097764 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:34.099909 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.101296 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.103636 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.104485 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.104656 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.109146 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:34.109173 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:34.109203 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:34.109271 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:34.114468 (+ 5197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.116495 (+ 2027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.116508 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.116512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.116816 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.117378 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.117384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.117387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.118085 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:34.118537 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.120074 (+ 1537us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.120406 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:42:34.121251 19027 raft_consensus.cc:2258] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:42:34.122218 18716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:42:34.125187 19028 ts_tablet_manager.cc:1918] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:34.125437 19028 log.cc:1196] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/f3a903ef9d8c4457a248f72e9e1e4420
I20260927 18:42:34.126250 19028 ts_tablet_manager.cc:1939] T f3a903ef9d8c4457a248f72e9e1e4420 P 9dbd5308b3c147049c91a913afcfb9bb: Deleting consensus metadata
I20260927 18:42:34.128446 19026 ts_tablet_manager.cc:1918] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:42:34.128782 19026 log.cc:1196] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/f3a903ef9d8c4457a248f72e9e1e4420
I20260927 18:42:34.129302 18520 catalog_manager.cc:4841] TS 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239): tablet f3a903ef9d8c4457a248f72e9e1e4420 (table test-table [id=eaa78a3f51674f3c8ab3a9a09cc0f7d7]) successfully deleted
I20260927 18:42:34.129542 19026 ts_tablet_manager.cc:1939] T f3a903ef9d8c4457a248f72e9e1e4420 P 3072bde1acd14db19da1948d5b55c599: Deleting consensus metadata
I20260927 18:42:34.131723 18520 catalog_manager.cc:4841] TS 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393): tablet f3a903ef9d8c4457a248f72e9e1e4420 (table test-table [id=eaa78a3f51674f3c8ab3a9a09cc0f7d7]) successfully deleted
I20260927 18:42:34.235844 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 18573
I20260927 18:42:34.255858 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 18725
W20260927 18:42:34.255875 18523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.95.65:39273 (error 108)
I20260927 18:42:34.258001 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.257464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39273 (local address 127.0.0.1:49904)
0927 18:42:34.257694 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.257831 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 18:42:34.258230 18523 catalog_manager.cc:4584] TS 235cf9bfacdd4c2fbdd25e22b4878177 (127.2.95.65:39273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3a903ef9d8c4457a248f72e9e1e4420: Network error: Client connection negotiation failed: client connection to 127.2.95.65:39273: connect: Connection refused (error 111)
I20260927 18:42:34.277415 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.256831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.2.95.66:37393)
0927 18:42:34.257058 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.257065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.257138 (+ 73us) server_negotiation.cc:407] Connection header received
0927 18:42:34.257376 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.257381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.257461 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.257595 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:34.259182 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.260038 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.262180 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.262623 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.262730 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.270279 (+ 7549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:34.270306 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:34.270330 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:34.270381 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:34.273725 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.274722 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.274732 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.274735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.274982 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.275445 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.275449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.275451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.276088 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:34.276414 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.276838 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.277067 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 18:42:34.278293 18848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:42:34.312408 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.311806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39273 (local address 127.0.0.1:49916)
0927 18:42:34.312087 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.312197 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 18:42:34.313704 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.313234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39273 (local address 127.0.0.1:49918)
0927 18:42:34.313403 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.313512 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 18:42:34.374969 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.374261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39273 (local address 127.0.0.1:49922)
0927 18:42:34.374581 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.374705 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260927 18:42:34.376617 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.376037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39273 (local address 127.0.0.1:49934)
0927 18:42:34.376253 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.376380 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 18:42:34.386667 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 18725
I20260927 18:42:34.404599 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 18857
I20260927 18:42:34.419786 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.405588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34852 (local address 127.2.95.67:36239)
0927 18:42:34.405828 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:34.405834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:34.405897 (+ 63us) server_negotiation.cc:407] Connection header received
0927 18:42:34.406101 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:34.406105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:34.406184 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:34.406278 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:34.407811 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.408652 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.410075 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:34.410507 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:34.410639 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:34.413441 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:34.413459 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:34.413483 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:34.413527 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:34.416734 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.417478 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.417487 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.417490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.417637 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:34.418053 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:34.418058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:34.418060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:34.418502 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:34.418809 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:34.419140 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 18:42:34.419387 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 18:42:34.420598 18980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:42:34.458143 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.457397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39273 (local address 127.0.0.1:49942)
0927 18:42:34.457747 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.457895 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 18:42:34.459590 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.459067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39273 (local address 127.0.0.1:49958)
0927 18:42:34.459311 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.459433 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 18:42:34.520895 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 18857
I20260927 18:42:34.538214 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 18503
I20260927 18:42:34.539711 18566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:42:34.672043 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:34.671396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:39273 (local address 127.0.0.1:49974)
0927 18:42:34.671610 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:34.671746 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.65:39273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 18:42:34.672013 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 18503
I20260927 18:42:54.696873 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:46029
--webserver_interface=127.2.95.126
--webserver_port=36761
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:46029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:42:55.099337 19034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:55.099746 19034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:55.099872 19034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:55.108114 19034 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 18:42:55.108265 19034 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 18:42:55.108305 19034 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 18:42:55.108363 19034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:42:55.108448 19034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:55.108553 19034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:55.108621 19034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:42:55.108687 19034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:42:55.120401 19034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:46029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:46029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=36761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:42:55.122305 19034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:55.124097 19034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:55.138871 19042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:55.139098 19040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:55.139375 19039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:55.141055 19034 server_base.cc:1034] running on GCE node
I20260927 18:42:55.142161 19034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:55.144112 19034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:55.145381 19034 hybrid_clock.cc:648] HybridClock initialized: now 1790534575145306 us; error 47 us; skew 500 ppm
I20260927 18:42:55.145851 19034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:42:55 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:55.149722 19034 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:42:55.151757 19034 webserver.cc:460] Webserver started at http://127.2.95.126:36761/ using document root <none> and password file <none>
I20260927 18:42:55.152670 19034 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:55.152796 19034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:55.159963 19034 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 18:42:55.164104 19049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:55.166126 19034 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 18:42:55.166371 19034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "101f14c6069a4d7c9cd66a5a76f0cf79"
format_stamp: "Formatted at 2026-09-27 18:42:08 on dist-test-slave-9jll"
server_key: "7063e14f1678e7ffa27772f86ea196a8"
server_key_iv: "ad747a4d8336dc0362d4e97a92dc06e8"
server_key_version: "encryptionkey@0"
I20260927 18:42:55.167644 19034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:55.192507 19034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:55.197639 19034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:55.198007 19034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:55.220413 19034 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:46029
I20260927 18:42:55.220453 19100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:46029 every 8 connection(s)
I20260927 18:42:55.222433 19034 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:42:55.227767 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 19034
I20260927 18:42:55.228853 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:39273
--local_ip_for_outbound_sockets=127.2.95.65
--tserver_master_addrs=127.2.95.126:46029
--webserver_port=38547
--webserver_interface=127.2.95.65
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:42:55.232185 19101 sys_catalog.cc:263] Verifying existing consensus state
I20260927 18:42:55.237141 19101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Bootstrap starting.
I20260927 18:42:55.267119 19101 log.cc:824] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:55.287824 19101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:42:55.288467 19101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Bootstrap complete.
I20260927 18:42:55.299118 19101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } }
I20260927 18:42:55.300309 19101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101f14c6069a4d7c9cd66a5a76f0cf79, State: Initialized, Role: FOLLOWER
I20260927 18:42:55.301138 19101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } }
I20260927 18:42:55.301393 19101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:42:55.301512 19101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:42:55.301736 19101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 2 FOLLOWER]: Advancing to term 3
I20260927 18:42:55.306885 19101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } }
I20260927 18:42:55.307489 19101 leader_election.cc:302] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 101f14c6069a4d7c9cd66a5a76f0cf79; no voters:
I20260927 18:42:55.308245 19101 leader_election.cc:288] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 18:42:55.308554 19105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 18:42:55.309521 19105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [term 3 LEADER]: Becoming Leader. State: Replica: 101f14c6069a4d7c9cd66a5a76f0cf79, State: Running, Role: LEADER
I20260927 18:42:55.310359 19105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } }
I20260927 18:42:55.310693 19101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:42:55.313812 19107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 101f14c6069a4d7c9cd66a5a76f0cf79. Latest consensus state: current_term: 3 leader_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } } }
I20260927 18:42:55.314424 19107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:55.316123 19106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101f14c6069a4d7c9cd66a5a76f0cf79" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 46029 } } }
I20260927 18:42:55.316548 19106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79 [sys.catalog]: This master's current role is: LEADER
I20260927 18:42:55.323367 19113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:42:55.335363 19113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:42:55.337234 19113 catalog_manager.cc:1259] Loaded cluster ID: 5540ac4bc99141ddae91ae3caf24cdeb
I20260927 18:42:55.337522 19113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:42:55.342543 19113 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:42:55.344905 19113 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Loaded TSK: 0
I20260927 18:42:55.345208 19113 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 101f14c6069a4d7c9cd66a5a76f0cf79: Loaded TSK: 1
F20260927 18:42:55.346625 19113 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f58d8b029ad google::LogMessage::Fail() at ??:0
@ 0x7f58d8b0588a google::LogMessage::SendToLog() at ??:0
@ 0x7f58d8b024de google::LogMessage::Flush() at ??:0
@ 0x7f58d8b03a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f58dc8667d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f58dc85573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f58ee5311c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f58ee530783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f58ee5bc2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f58ee5bc0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f58ee6065e9 std::function<>::operator()() at ??:0
@ 0x7f58ee534cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f58ee532c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f58ee5ba610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f58ee5ba439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f58db74e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f58db75191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f58db751799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f58db72ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f58dea066db start_thread at ??:0
@ 0x7f58d799c61f clone at ??:0
W20260927 18:42:55.706719 19103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:55.707116 19103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:55.707262 19103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:55.715636 19103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:55.715840 19103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:55.716146 19103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:42:55.727770 19103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:39273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=38547
--tserver_master_addrs=127.2.95.126:46029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:42:55.729610 19103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:55.731428 19103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:55.748018 19148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:55.748113 19151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:55.747972 19149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:55.749038 19103 server_base.cc:1034] running on GCE node
I20260927 18:42:55.749891 19103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:55.751118 19103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:55.752351 19103 hybrid_clock.cc:648] HybridClock initialized: now 1790534575752270 us; error 74 us; skew 500 ppm
I20260927 18:42:55.752734 19103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:42:55 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:55.756574 19103 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:42:55.758472 19103 webserver.cc:460] Webserver started at http://127.2.95.65:38547/ using document root <none> and password file <none>
I20260927 18:42:55.759363 19103 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:55.759537 19103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:55.766521 19103 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 18:42:55.770211 19158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:55.772076 19103 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:42:55.772303 19103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "235cf9bfacdd4c2fbdd25e22b4878177"
format_stamp: "Formatted at 2026-09-27 18:42:08 on dist-test-slave-9jll"
server_key: "b44fc9f2534c3ae62c1ef2c35de610a6"
server_key_iv: "130e8e90fd3699c96fbacfef5560595a"
server_key_version: "encryptionkey@0"
I20260927 18:42:55.773756 19103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:55.820477 19103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:55.826561 19103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:55.826905 19103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:55.828456 19103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:55.830925 19165 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 18:42:55.838506 19103 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 18:42:55.838694 19103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260927 18:42:55.838876 19103 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 18:42:55.843175 19103 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 18:42:55.843293 19103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20260927 18:42:55.843917 19165 tablet_bootstrap.cc:492] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: Bootstrap starting.
I20260927 18:42:55.855543 19165 log.cc:824] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: Log is configured to *not* fsync() on all Append() calls
I20260927 18:42:55.880024 19165 tablet_bootstrap.cc:492] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 18:42:55.881071 19165 tablet_bootstrap.cc:492] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: Bootstrap complete.
I20260927 18:42:55.881881 19165 ts_tablet_manager.cc:1397] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: Time spent bootstrapping tablet: real 0.039s user 0.032s sys 0.005s
I20260927 18:42:55.891021 19103 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:39273
I20260927 18:42:55.891049 19272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:39273 every 8 connection(s)
I20260927 18:42:55.893162 19103 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:42:55.895174 19165 raft_consensus.cc:348] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:55.896512 19165 raft_consensus.cc:729] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235cf9bfacdd4c2fbdd25e22b4878177, State: Initialized, Role: FOLLOWER
I20260927 18:42:55.897413 19165 consensus_queue.cc:260] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:55.899116 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 19103
I20260927 18:42:55.900003 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:37393
--local_ip_for_outbound_sockets=127.2.95.66
--tserver_master_addrs=127.2.95.126:46029
--webserver_port=37607
--webserver_interface=127.2.95.66
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:42:55.901111 19165 ts_tablet_manager.cc:1428] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
I20260927 18:42:55.901996 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:55.897267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:40583)
0927 18:42:55.899582 (+ 2315us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:55.900028 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1120,"thread_start_us":784,"threads_started":1}
W20260927 18:42:55.902938 19273 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:46029 (0 consecutive failures): Network error: Failed to ping master at 127.2.95.126:46029: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
I20260927 18:42:55.904084 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:55.903612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:48837)
0927 18:42:55.903809 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:55.903911 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 18:42:55.905784 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:55.904683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:56155)
0927 18:42:55.905309 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:55.905480 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":298,"threads_started":1}
W20260927 18:42:55.906067 19273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:42:56.308143 19278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:56.308542 19278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:56.308651 19278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:56.317413 19278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:56.317606 19278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:56.317776 19278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:42:56.332010 19278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:37393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=37607
--tserver_master_addrs=127.2.95.126:46029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:42:56.333837 19278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:56.335511 19278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:56.351811 19284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:56.352171 19286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:56.351799 19283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:56.352614 19278 server_base.cc:1034] running on GCE node
I20260927 18:42:56.353403 19278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:56.354656 19278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:56.355937 19278 hybrid_clock.cc:648] HybridClock initialized: now 1790534576355780 us; error 85 us; skew 500 ppm
I20260927 18:42:56.356302 19278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:42:56 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534576, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:56.360276 19278 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:42:56.362284 19278 webserver.cc:460] Webserver started at http://127.2.95.66:37607/ using document root <none> and password file <none>
I20260927 18:42:56.363137 19278 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:56.363253 19278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:56.370004 19278 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 18:42:56.373759 19293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:56.375613 19278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:42:56.375806 19278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "3072bde1acd14db19da1948d5b55c599"
format_stamp: "Formatted at 2026-09-27 18:42:09 on dist-test-slave-9jll"
server_key: "ca9d3932df951d780365933fc4de9c00"
server_key_iv: "04ec95db9a8f1bd716ec40b3d4ab5f99"
server_key_version: "encryptionkey@0"
I20260927 18:42:56.377017 19278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:56.405836 19278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:56.411513 19278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:56.411829 19278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:56.413024 19278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:56.414700 19278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:56.414804 19278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:56.414929 19278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:56.415011 19278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:56.451227 19278 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:37393
I20260927 18:42:56.451298 19405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:37393 every 8 connection(s)
I20260927 18:42:56.453027 19278 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:42:56.454355 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 19278
I20260927 18:42:56.455247 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:36239
--local_ip_for_outbound_sockets=127.2.95.67
--tserver_master_addrs=127.2.95.126:46029
--webserver_port=42897
--webserver_interface=127.2.95.67
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:42:56.459831 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:56.457240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:34687)
0927 18:42:56.458308 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:56.458646 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":443,"threads_started":1}
W20260927 18:42:56.460664 19406 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:46029 (0 consecutive failures): Network error: Failed to ping master at 127.2.95.126:46029: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
I20260927 18:42:56.461730 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:56.461313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:58263)
0927 18:42:56.461485 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:56.461565 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 18:42:56.462761 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:56.462386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:43981)
0927 18:42:56.462551 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:56.462632 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 18:42:56.463037 19406 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 18:42:56.854061 19409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:42:56.854446 19409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:42:56.854573 19409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:42:56.862844 19409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:42:56.862998 19409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:42:56.863178 19409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:42:56.874488 19409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:36239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=42897
--tserver_master_addrs=127.2.95.126:46029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:42:56.876343 19409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:42:56.878110 19409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:42:56.893473 19414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:56.893446 19415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:42:56.893970 19417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:42:56.894779 19409 server_base.cc:1034] running on GCE node
I20260927 18:42:56.895504 19409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:42:56.896715 19409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:42:56.897926 19409 hybrid_clock.cc:648] HybridClock initialized: now 1790534576897855 us; error 73 us; skew 500 ppm
I20260927 18:42:56.898293 19409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:42:56 dist-test-slave-9jll krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534576, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:42:56.901907 19409 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:42:56.903671 19409 webserver.cc:460] Webserver started at http://127.2.95.67:42897/ using document root <none> and password file <none>
I20260927 18:42:56.904474 19409 fs_manager.cc:359] Metadata directory not provided
I20260927 18:42:56.904587 19409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:42:56.907706 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:56.906803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:52367)
0927 18:42:56.907337 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:56.907440 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":263,"threads_started":1}
I20260927 18:42:56.911455 19409 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 18:42:56.914893 19425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:56.916653 19409 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:42:56.916815 19409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "9dbd5308b3c147049c91a913afcfb9bb"
format_stamp: "Formatted at 2026-09-27 18:42:10 on dist-test-slave-9jll"
server_key: "ce7a1c458719deed0c758790f63168f4"
server_key_iv: "8e051d2c7caeac7d6eb4e0c8c02cea09"
server_key_version: "encryptionkey@0"
I20260927 18:42:56.918028 19409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:42:56.944123 19409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:42:56.949424 19409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:42:56.949754 19409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:42:56.950958 19409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:42:56.952579 19409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:42:56.952682 19409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:56.952800 19409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:42:56.952878 19409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:42:56.989070 19409 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:36239
I20260927 18:42:56.989135 19537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:36239 every 8 connection(s)
I20260927 18:42:56.990911 19409 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:42:56.996546 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:56.994373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:42257)
0927 18:42:56.995217 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:56.995474 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":314,"threads_started":1}
W20260927 18:42:56.997365 19538 heartbeater.cc:644] Failed to heartbeat to 127.2.95.126:46029 (0 consecutive failures): Network error: Failed to ping master at 127.2.95.126:46029: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
I20260927 18:42:56.998206 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 19409
I20260927 18:42:56.998417 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:56.998014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:51447)
0927 18:42:56.998181 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:56.998265 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 18:42:56.999559 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:56.999121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:41639)
0927 18:42:56.999284 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:56.999386 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 18:42:56.999923 19538 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 18:42:57.465169 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:57.463940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:49183)
0927 18:42:57.464632 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:57.464781 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":294,"threads_started":1}
I20260927 18:42:57.617542 19543 raft_consensus.cc:482] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:57.618046 19543 raft_consensus.cc:504] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:57.619843 19543 leader_election.cc:288] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393), 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239)
Sep 27 18:42:57 dist-test-slave-9jll krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
Sep 27 18:42:57 dist-test-slave-9jll krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534575, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:42:57.648278 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:57.621275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:42445 (local address 127.2.95.67:36239)
0927 18:42:57.622618 (+ 1343us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:57.622639 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:57.623272 (+ 633us) server_negotiation.cc:407] Connection header received
0927 18:42:57.625274 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:57.625287 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:57.625903 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:57.626302 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:57.628827 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.630866 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.633127 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.633942 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.634160 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:57.638614 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:57.638625 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:57.638645 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:57.638714 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:57.643254 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:57.644595 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:57.644606 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:57.644614 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:57.644895 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:57.645443 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:57.645448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:57.645450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:57.646182 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:57.646393 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:57.647313 (+ 920us) server_negotiation.cc:299] Negotiation successful
0927 18:42:57.647798 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":1019,"threads_started":1}
I20260927 18:42:57.648286 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:57.620648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:44887 (local address 127.2.95.66:37393)
0927 18:42:57.623205 (+ 2557us) server_negotiation.cc:206] Beginning negotiation
0927 18:42:57.623222 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:42:57.623274 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:42:57.625212 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:42:57.625224 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:42:57.625829 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:42:57.626232 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:42:57.628743 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.630863 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.633218 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.634071 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.634306 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:57.638939 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:42:57.638953 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:42:57.638970 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:42:57.639029 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:42:57.643254 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:57.644389 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:57.644400 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:57.644408 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:57.644683 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:42:57.645246 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:42:57.645253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:42:57.645257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:42:57.646158 (+ 901us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:42:57.646375 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 18:42:57.647351 (+ 976us) server_negotiation.cc:299] Negotiation successful
0927 18:42:57.647843 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2332,"thread_start_us":2067,"threads_started":1}
I20260927 18:42:57.648345 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:57.620080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:37393 (local address 127.2.95.65:44887)
0927 18:42:57.620709 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:57.620759 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:57.621607 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:57.626496 (+ 4889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:57.626511 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:57.627541 (+ 1030us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:57.628493 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:57.628525 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.631148 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.631154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:57.632954 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:57.632974 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.634335 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.634342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:57.634557 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:57.635351 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:57.635383 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:57.638559 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:57.643516 (+ 4957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:57.643530 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:57.643548 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:57.644219 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:57.644815 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:57.644820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:57.644823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:57.645038 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:57.646334 (+ 1296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:57.646343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:57.646714 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 18:42:57.647029 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 18:42:57.647755 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":254,"spinlock_wait_cycles":3840,"thread_start_us":313,"threads_started":1}
I20260927 18:42:57.648780 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:57.620902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:36239 (local address 127.2.95.65:42445)
0927 18:42:57.623089 (+ 2187us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:57.623110 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:42:57.623264 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:42:57.626501 (+ 3237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:42:57.626512 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:42:57.627541 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:42:57.628504 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:57.628529 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.631146 (+ 2617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.631151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:57.632945 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:42:57.632968 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:42:57.634154 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:42:57.634161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:42:57.634443 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:42:57.635220 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:42:57.635248 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:42:57.638244 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:42:57.643523 (+ 5279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:57.643533 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:57.643548 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:57.644284 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:57.645041 (+ 757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:42:57.645047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:42:57.645051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:42:57.645307 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:42:57.646352 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:42:57.646357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:42:57.646701 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 18:42:57.647028 (+ 327us) client_negotiation.cc:240] Negotiation successful
0927 18:42:57.647970 (+ 942us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2049,"mutex_wait_us":428,"spinlock_wait_cycles":71040,"thread_start_us":1878,"threads_started":1}
W20260927 18:42:57.652900 19159 leader_election.cc:341] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239): Not found: Tablet not found: f3a903ef9d8c4457a248f72e9e1e4420
W20260927 18:42:57.653263 19159 leader_election.cc:341] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393): Not found: Tablet not found: f3a903ef9d8c4457a248f72e9e1e4420
I20260927 18:42:57.653403 19159 leader_election.cc:302] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 235cf9bfacdd4c2fbdd25e22b4878177; no voters: 3072bde1acd14db19da1948d5b55c599, 9dbd5308b3c147049c91a913afcfb9bb
I20260927 18:42:57.653882 19543 raft_consensus.cc:2726] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:42:57.909478 19546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:57.908957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:48411)
0927 18:42:57.909187 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:57.909320 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 18:42:58.002125 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:58.000804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:60957)
0927 18:42:58.001489 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:58.001754 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":308,"threads_started":1}
I20260927 18:42:58.467553 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:58.466557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:45243)
0927 18:42:58.467209 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:58.467329 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":362,"threads_started":1}
I20260927 18:42:58.912055 19550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:58.910929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:52833)
0927 18:42:58.911669 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:58.911799 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":317,"threads_started":1}
I20260927 18:42:59.004222 19551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:59.003357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:58993)
0927 18:42:59.003927 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:59.004030 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":302,"threads_started":1}
I20260927 18:42:59.398711 19552 raft_consensus.cc:482] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:42:59.398952 19552 raft_consensus.cc:504] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:42:59.399856 19552 leader_election.cc:288] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393), 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239)
W20260927 18:42:59.401332 19159 leader_election.cc:341] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393): Not found: Tablet not found: f3a903ef9d8c4457a248f72e9e1e4420
W20260927 18:42:59.401576 19159 leader_election.cc:341] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239): Not found: Tablet not found: f3a903ef9d8c4457a248f72e9e1e4420
I20260927 18:42:59.401722 19159 leader_election.cc:302] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 235cf9bfacdd4c2fbdd25e22b4878177; no voters: 3072bde1acd14db19da1948d5b55c599, 9dbd5308b3c147049c91a913afcfb9bb
I20260927 18:42:59.402067 19552 raft_consensus.cc:2726] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:42:59.470248 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:59.469032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:59441)
0927 18:42:59.469915 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:59.470034 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":348,"threads_started":1}
I20260927 18:42:59.914232 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:42:59.913377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:34187)
0927 18:42:59.913946 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 18:42:59.914047 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":318,"threads_started":1}
I20260927 18:43:00.007375 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:00.006015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:55905)
0927 18:43:00.006819 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:00.007082 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":394,"threads_started":1}
I20260927 18:43:00.472831 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:00.471818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:33563)
0927 18:43:00.472483 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:00.472582 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":308,"threads_started":1}
I20260927 18:43:00.916495 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:00.915562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:36811)
0927 18:43:00.916157 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:00.916265 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":291,"threads_started":1}
I20260927 18:43:01.010221 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:01.009118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:35087)
0927 18:43:01.009791 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:01.009996 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":299,"threads_started":1}
I20260927 18:43:01.474910 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:01.473971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:48671)
0927 18:43:01.474590 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:01.474690 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":333,"threads_started":1}
I20260927 18:43:01.904214 19560 raft_consensus.cc:482] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:43:01.904458 19560 raft_consensus.cc:504] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:43:01.905179 19560 leader_election.cc:288] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393), 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239)
W20260927 18:43:01.906519 19159 leader_election.cc:341] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393): Not found: Tablet not found: f3a903ef9d8c4457a248f72e9e1e4420
W20260927 18:43:01.906734 19159 leader_election.cc:341] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239): Not found: Tablet not found: f3a903ef9d8c4457a248f72e9e1e4420
I20260927 18:43:01.906831 19159 leader_election.cc:302] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 235cf9bfacdd4c2fbdd25e22b4878177; no voters: 3072bde1acd14db19da1948d5b55c599, 9dbd5308b3c147049c91a913afcfb9bb
I20260927 18:43:01.907087 19560 raft_consensus.cc:2726] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:43:01.918784 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:01.917553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:39169)
0927 18:43:01.918345 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:01.918530 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":421,"threads_started":1}
I20260927 18:43:02.012480 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:02.011529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:36949)
0927 18:43:02.012186 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:02.012303 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":341,"threads_started":1}
I20260927 18:43:02.477367 19563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:02.476221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:34443)
0927 18:43:02.476994 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:02.477120 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":411,"threads_started":1}
I20260927 18:43:02.921228 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:02.920187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:35809)
0927 18:43:02.920811 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:02.920930 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":315,"threads_started":1}
I20260927 18:43:03.015038 19565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:03.013972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:48973)
0927 18:43:03.014598 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:03.014814 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":317,"threads_started":1}
I20260927 18:43:03.480019 19566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:03.478783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:46661)
0927 18:43:03.479588 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:03.479743 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":375,"threads_started":1}
I20260927 18:43:03.578321 19567 raft_consensus.cc:482] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:43:03.578604 19567 raft_consensus.cc:504] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:43:03.579409 19567 leader_election.cc:288] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393), 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239)
W20260927 18:43:03.580785 19159 leader_election.cc:341] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239): Not found: Tablet not found: f3a903ef9d8c4457a248f72e9e1e4420
W20260927 18:43:03.581070 19159 leader_election.cc:341] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393): Not found: Tablet not found: f3a903ef9d8c4457a248f72e9e1e4420
I20260927 18:43:03.581159 19159 leader_election.cc:302] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 235cf9bfacdd4c2fbdd25e22b4878177; no voters: 3072bde1acd14db19da1948d5b55c599, 9dbd5308b3c147049c91a913afcfb9bb
I20260927 18:43:03.581465 19567 raft_consensus.cc:2726] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:43:03.923317 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:03.922453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:60809)
0927 18:43:03.922983 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:03.923105 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":286,"threads_started":1}
I20260927 18:43:04.017864 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:04.016299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:40789)
0927 18:43:04.017164 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:04.017313 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":354,"threads_started":1}
I20260927 18:43:04.482410 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:04.481294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:58055)
0927 18:43:04.482016 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:04.482151 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":298,"threads_started":1}
I20260927 18:43:04.925294 19571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:04.924320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:52743)
0927 18:43:04.924972 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:04.925091 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":308,"threads_started":1}
I20260927 18:43:05.020028 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:05.019107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:59029)
0927 18:43:05.019686 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:05.019806 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":283,"threads_started":1}
I20260927 18:43:05.484433 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:05.483465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:51167)
0927 18:43:05.484131 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:05.484243 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":329,"threads_started":1}
I20260927 18:43:05.927908 19574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:05.926588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:50461)
0927 18:43:05.927433 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:05.927660 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":442,"threads_started":1}
I20260927 18:43:06.022620 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:06.021536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:57435)
0927 18:43:06.022324 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:06.022445 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":423,"threads_started":1}
I20260927 18:43:06.486761 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:06.485791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:36089)
0927 18:43:06.486406 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:06.486536 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":308,"threads_started":1}
I20260927 18:43:06.930565 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:06.929126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:49747)
0927 18:43:06.930135 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:06.930294 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":374,"threads_started":1}
I20260927 18:43:07.024781 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:07.023856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:43623)
0927 18:43:07.024458 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:07.024566 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":325,"threads_started":1}
I20260927 18:43:07.489169 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:07.488067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:53975)
0927 18:43:07.488785 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:07.488930 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":361,"threads_started":1}
I20260927 18:43:07.933506 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:07.932202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:46581)
0927 18:43:07.933085 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:07.933247 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":578,"threads_started":1}
I20260927 18:43:08.026983 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:08.025863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:47197)
0927 18:43:08.026551 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:08.026703 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":316,"threads_started":1}
I20260927 18:43:08.491660 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:08.490639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:50733)
0927 18:43:08.491308 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:08.491426 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":342,"threads_started":1}
I20260927 18:43:08.935849 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:08.934896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:42947)
0927 18:43:08.935503 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:08.935611 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":321,"threads_started":1}
I20260927 18:43:09.029297 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.028373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:47023)
0927 18:43:09.028985 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:09.029082 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":318,"threads_started":1}
I20260927 18:43:09.493993 19585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.492999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:43271)
0927 18:43:09.493649 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:09.493806 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":314,"threads_started":1}
I20260927 18:43:09.938446 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:09.937331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:45239)
0927 18:43:09.938043 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:09.938220 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":421,"threads_started":1}
I20260927 18:43:10.031697 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:10.030741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:52063)
0927 18:43:10.031343 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:10.031479 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":291,"threads_started":1}
I20260927 18:43:10.496433 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:10.495378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:50769)
0927 18:43:10.496048 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:10.496160 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":329,"threads_started":1}
I20260927 18:43:10.761113 19589 raft_consensus.cc:482] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:43:10.761358 19589 raft_consensus.cc:504] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3072bde1acd14db19da1948d5b55c599" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37393 } } peers { permanent_uuid: "235cf9bfacdd4c2fbdd25e22b4878177" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 39273 } } peers { permanent_uuid: "9dbd5308b3c147049c91a913afcfb9bb" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 36239 } }
I20260927 18:43:10.762190 19589 leader_election.cc:288] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393), 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239)
W20260927 18:43:10.763556 19159 leader_election.cc:341] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dbd5308b3c147049c91a913afcfb9bb (127.2.95.67:36239): Not found: Tablet not found: f3a903ef9d8c4457a248f72e9e1e4420
W20260927 18:43:10.763902 19159 leader_election.cc:341] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3072bde1acd14db19da1948d5b55c599 (127.2.95.66:37393): Not found: Tablet not found: f3a903ef9d8c4457a248f72e9e1e4420
I20260927 18:43:10.764020 19159 leader_election.cc:302] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 235cf9bfacdd4c2fbdd25e22b4878177; no voters: 3072bde1acd14db19da1948d5b55c599, 9dbd5308b3c147049c91a913afcfb9bb
I20260927 18:43:10.764325 19589 raft_consensus.cc:2726] T f3a903ef9d8c4457a248f72e9e1e4420 P 235cf9bfacdd4c2fbdd25e22b4878177 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 18:43:10.941072 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:10.940060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:58829)
0927 18:43:10.940646 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:10.940769 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":291,"threads_started":1}
I20260927 18:43:11.033910 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:11.032883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:37703)
0927 18:43:11.033494 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:11.033600 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":302,"threads_started":1}
I20260927 18:43:11.499176 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:11.498197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.66:34835)
0927 18:43:11.498821 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:11.498931 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":344,"threads_started":1}
I20260927 18:43:11.943221 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:11.942267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.65:53503)
0927 18:43:11.942884 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:11.943004 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":293,"threads_started":1}
I20260927 18:43:12.009212 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 19103
I20260927 18:43:12.032425 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:12.010710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.2.95.65:39273)
0927 18:43:12.011432 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:12.011440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:12.011473 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:43:12.011546 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:12.011553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:12.011801 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:12.011971 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:12.013584 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:12.014769 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:12.016233 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:12.016825 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:12.016934 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:12.025393 (+ 8459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:12.025414 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:12.025433 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:12.025488 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:12.029042 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:12.030012 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:12.030020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:12.030023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:12.030190 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:12.030766 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:12.030770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:12.030773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:12.031184 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:12.031455 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:12.031868 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 18:43:12.032101 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":360,"threads_started":1}
I20260927 18:43:12.034303 19268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:12.036448 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:12.035415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:46029 (local address 127.2.95.67:51115)
0927 18:43:12.036051 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:12.036204 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.126:46029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":333,"threads_started":1}
I20260927 18:43:12.138008 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 19103
I20260927 18:43:12.157341 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 19278
I20260927 18:43:12.174141 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:12.158374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.2.95.66:37393)
0927 18:43:12.159083 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:12.159089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:12.159112 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:43:12.159180 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:12.159184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:12.159257 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:12.159379 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:12.161081 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:12.162112 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:12.163749 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:12.164230 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:12.164360 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:12.167551 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:12.167566 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:12.167572 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:12.167620 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:12.170779 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:12.171813 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:12.171818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:12.171821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:12.172004 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:12.172450 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:12.172454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:12.172456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:12.172865 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:12.173112 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:12.173466 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 18:43:12.173801 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":410,"threads_started":1}
I20260927 18:43:12.175071 19401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:12.271574 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 19278
I20260927 18:43:12.290403 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 19409
I20260927 18:43:12.306378 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:12.291563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.2.95.67:36239)
0927 18:43:12.292322 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:12.292330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:12.292354 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:43:12.292420 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:12.292425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:12.292512 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:12.292623 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:12.294294 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:12.295207 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:12.296634 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:12.297115 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:12.297269 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:12.300263 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:12.300269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:12.300273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:12.300318 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:12.303287 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:12.304102 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:12.304107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:12.304135 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:12.304306 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:12.304797 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:12.304805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:12.304808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:12.305144 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:12.305378 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:12.305802 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 18:43:12.306048 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":385,"threads_started":1}
I20260927 18:43:12.307315 19533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:12.401933 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 19409
2026-09-27T18:43:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66237 ms)
[----------] 30 tests from SecurityITest (307640 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:12 dist-test-slave-9jll krb5kdc[19603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:12 dist-test-slave-9jll krb5kdc[19603](info): set up 2 sockets
Sep 27 18:43:12 dist-test-slave-9jll krb5kdc[19603](info): commencing operation
krb5kdc: starting...
W20260927 18:43:13.450264 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.009s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:13 dist-test-slave-9jll krb5kdc[19603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:43:13.750976 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:37217
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:37217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:14.166837 19616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:14.167225 19616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:14.167366 19616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:14.175468 19616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:14.175611 19616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:14.175674 19616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:14.175768 19616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:14.175825 19616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:14.187088 19616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:37217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:37217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:14.188865 19616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:14.190555 19616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:14.204798 19624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:14.204931 19621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:14.204917 19622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:14.206041 19616 server_base.cc:1034] running on GCE node
I20260927 18:43:14.207056 19616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:14.208961 19616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:14.210202 19616 hybrid_clock.cc:648] HybridClock initialized: now 1790534594210140 us; error 41 us; skew 500 ppm
I20260927 18:43:14.210665 19616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:14 dist-test-slave-9jll krb5kdc[19603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:14.214738 19616 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:43:14.216884 19616 webserver.cc:460] Webserver started at http://127.2.95.126:43969/ using document root <none> and password file <none>
I20260927 18:43:14.217836 19616 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:14.217983 19616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:14.218346 19616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:14.222188 19616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "cb8ec27a5f994c60a0d629e68b30d20f"
format_stamp: "Formatted at 2026-09-27 18:43:14 on dist-test-slave-9jll"
server_key: "9fa82d3d2538627256ab271f7ed2a865"
server_key_iv: "42d9b597ebcbd8c8f330590cae6ab6df"
server_key_version: "encryptionkey@0"
I20260927 18:43:14.223093 19616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "cb8ec27a5f994c60a0d629e68b30d20f"
format_stamp: "Formatted at 2026-09-27 18:43:14 on dist-test-slave-9jll"
server_key: "9fa82d3d2538627256ab271f7ed2a865"
server_key_iv: "42d9b597ebcbd8c8f330590cae6ab6df"
server_key_version: "encryptionkey@0"
I20260927 18:43:14.229451 19616 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 18:43:14.233649 19631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:14.235261 19616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:43:14.235447 19616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "cb8ec27a5f994c60a0d629e68b30d20f"
format_stamp: "Formatted at 2026-09-27 18:43:14 on dist-test-slave-9jll"
server_key: "9fa82d3d2538627256ab271f7ed2a865"
server_key_iv: "42d9b597ebcbd8c8f330590cae6ab6df"
server_key_version: "encryptionkey@0"
I20260927 18:43:14.235662 19616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:14.264622 19616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:14.269948 19616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:14.270280 19616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:14.291533 19616 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:37217
I20260927 18:43:14.291548 19682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:37217 every 8 connection(s)
I20260927 18:43:14.293545 19616 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:14.298859 19683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:14.303978 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 19616
I20260927 18:43:14.304267 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:14.305179 2429 external_mini_cluster.cc:1428] Setting key b58207170f1248587c810d3554f8824f
I20260927 18:43:14.315006 19683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f: Bootstrap starting.
I20260927 18:43:14.319824 19683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:14.321472 19683 log.cc:824] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:14.325693 19683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f: No bootstrap required, opened a new log
Sep 27 18:43:14 dist-test-slave-9jll krb5kdc[19603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:43:14.337220 19683 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8ec27a5f994c60a0d629e68b30d20f" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37217 } }
I20260927 18:43:14.337816 19683 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:14.337940 19683 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8ec27a5f994c60a0d629e68b30d20f, State: Initialized, Role: FOLLOWER
I20260927 18:43:14.338454 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:14.308152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.2.95.126:37217)
0927 18:43:14.309238 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:14.309271 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:14.309661 (+ 390us) server_negotiation.cc:407] Connection header received
0927 18:43:14.312793 (+ 3132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:14.312835 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:14.313577 (+ 742us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:14.314220 (+ 643us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:14.316130 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:14.318378 (+ 2248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:14.320004 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:14.321164 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:14.321371 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:14.328268 (+ 6897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:14.328283 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:14.328315 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:14.328412 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:14.332595 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:14.333708 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:14.333719 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:14.333731 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:14.334051 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:14.334494 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:14.334497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:14.334499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:14.335312 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:14.335589 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:14.336217 (+ 628us) server_negotiation.cc:299] Negotiation successful
0927 18:43:14.336836 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":447,"threads_started":1}
I20260927 18:43:14.338793 19683 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8ec27a5f994c60a0d629e68b30d20f" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37217 } }
I20260927 18:43:14.339151 19683 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:14.339354 19683 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:14.339576 19683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:14.342736 19683 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8ec27a5f994c60a0d629e68b30d20f" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37217 } }
I20260927 18:43:14.343318 19683 leader_election.cc:302] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8ec27a5f994c60a0d629e68b30d20f; no voters:
I20260927 18:43:14.344030 19683 leader_election.cc:288] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:14.344194 19688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:14.344904 19688 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [term 1 LEADER]: Becoming Leader. State: Replica: cb8ec27a5f994c60a0d629e68b30d20f, State: Running, Role: LEADER
I20260927 18:43:14.345505 19688 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8ec27a5f994c60a0d629e68b30d20f" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37217 } }
I20260927 18:43:14.346539 19683 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:14.348758 19689 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb8ec27a5f994c60a0d629e68b30d20f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8ec27a5f994c60a0d629e68b30d20f" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37217 } } }
I20260927 18:43:14.349267 19689 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:14.351531 19690 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb8ec27a5f994c60a0d629e68b30d20f. Latest consensus state: current_term: 1 leader_uuid: "cb8ec27a5f994c60a0d629e68b30d20f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8ec27a5f994c60a0d629e68b30d20f" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 37217 } } }
I20260927 18:43:14.351927 19690 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:14.352770 19697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:14.359292 19697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:14.370566 19697 catalog_manager.cc:1347] Generated new cluster ID: 16d81fca845f4949806cf500a638a47e
I20260927 18:43:14.370744 19697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:14.400624 19697 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:14.402350 19697 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:14.410140 19697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f: Generated new TSK 0
I20260927 18:43:14.411264 19697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:43:14.497989 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37217
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:14.906572 19709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:14.907001 19709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:14.907166 19709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:14.916162 19709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:14.916328 19709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:14.916503 19709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:43:14.928447 19709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:14.930346 19709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:14.932126 19709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:14.947564 19717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:14.947592 19714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:14.947495 19715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:14.948521 19709 server_base.cc:1034] running on GCE node
I20260927 18:43:14.949340 19709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:14.950575 19709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:14.951798 19709 hybrid_clock.cc:648] HybridClock initialized: now 1790534594951733 us; error 57 us; skew 500 ppm
I20260927 18:43:14.952169 19709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:14 dist-test-slave-9jll krb5kdc[19603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:14.956192 19709 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:43:14.958148 19709 webserver.cc:460] Webserver started at http://127.2.95.65:38213/ using document root <none> and password file <none>
I20260927 18:43:14.959033 19709 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:14.959194 19709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:14.959569 19709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:14.963261 19709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d"
format_stamp: "Formatted at 2026-09-27 18:43:14 on dist-test-slave-9jll"
server_key: "e27f6833b32394167200d63722aa4301"
server_key_iv: "704f7dac3bac0999a5a8c9202f4e63f5"
server_key_version: "encryptionkey@0"
I20260927 18:43:14.964139 19709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d"
format_stamp: "Formatted at 2026-09-27 18:43:14 on dist-test-slave-9jll"
server_key: "e27f6833b32394167200d63722aa4301"
server_key_iv: "704f7dac3bac0999a5a8c9202f4e63f5"
server_key_version: "encryptionkey@0"
I20260927 18:43:14.970572 19709 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 18:43:14.975063 19724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:14.976851 19709 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:43:14.977052 19709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d"
format_stamp: "Formatted at 2026-09-27 18:43:14 on dist-test-slave-9jll"
server_key: "e27f6833b32394167200d63722aa4301"
server_key_iv: "704f7dac3bac0999a5a8c9202f4e63f5"
server_key_version: "encryptionkey@0"
I20260927 18:43:14.977295 19709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:14.997642 19709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:15.003396 19709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:15.003769 19709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:15.005183 19709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:15.007046 19709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:15.007138 19709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:15.007275 19709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:15.007364 19709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:15.045379 19709 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:33225
I20260927 18:43:15.045418 19836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:33225 every 8 connection(s)
I20260927 18:43:15.047045 19709 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:43:15.055222 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 19709
I20260927 18:43:15.055470 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:43:15.056187 2429 external_mini_cluster.cc:1428] Setting key c85542199909be3c582afc1d0880692b
Sep 27 18:43:15 dist-test-slave-9jll krb5kdc[19603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:43:15.070014 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:15.050299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:34699 (local address 127.2.95.126:37217)
0927 18:43:15.051073 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:15.051078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:15.052011 (+ 933us) server_negotiation.cc:407] Connection header received
0927 18:43:15.054232 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:15.054238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:15.054336 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:15.054451 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:15.056376 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:15.057330 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:15.059062 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:15.059494 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:15.059616 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:15.063691 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:15.063696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:15.063700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:15.063741 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:15.066528 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:15.067357 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:15.067362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:15.067364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:15.067518 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:15.067961 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:15.067965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:15.067968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:15.068282 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:15.068428 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:15.069284 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 18:43:15.069570 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":342,"threads_started":1}
I20260927 18:43:15.071260 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:15.050917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37217 (local address 127.2.95.65:34699)
0927 18:43:15.051795 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:15.051853 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:15.053901 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:15.054754 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:15.054765 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:15.055421 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:15.056179 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:15.056200 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:15.057491 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:15.057496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:15.058908 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:15.058921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:15.059752 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:15.059760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:15.060032 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:15.060810 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:15.060840 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:15.063422 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:15.066683 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:15.066694 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:15.066715 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:15.067218 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:15.067627 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:15.067632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:15.067635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:15.067842 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:15.068407 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:15.068417 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:15.068865 (+ 448us) client_negotiation.cc:769] Sending connection context
0927 18:43:15.069153 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 18:43:15.069838 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":353,"threads_started":1}
I20260927 18:43:15.073086 19837 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37217
I20260927 18:43:15.073423 19837 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:15.074353 19837 heartbeater.cc:511] Master 127.2.95.126:37217 requested a full tablet report, sending...
I20260927 18:43:15.077111 19648 ts_manager.cc:194] Registered new tserver with Master: f4bf30acbb8d46e3a67d4ed6196aaf9d (127.2.95.65:33225)
I20260927 18:43:15.079537 19648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:34699
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:43:15.122403 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37217
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:15.530349 19843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:15.530752 19843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:15.530901 19843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:15.538955 19843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:15.539100 19843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:15.539258 19843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:43:15.550933 19843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:15.552677 19843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:15.554347 19843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:15.569862 19848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:15.569842 19849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:15.569979 19851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:15.570668 19843 server_base.cc:1034] running on GCE node
I20260927 18:43:15.571426 19843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:15.572664 19843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:15.573899 19843 hybrid_clock.cc:648] HybridClock initialized: now 1790534595573844 us; error 55 us; skew 500 ppm
I20260927 18:43:15.574252 19843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:15 dist-test-slave-9jll krb5kdc[19603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:15.577965 19843 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:43:15.579643 19843 webserver.cc:460] Webserver started at http://127.2.95.66:35919/ using document root <none> and password file <none>
I20260927 18:43:15.580442 19843 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:15.580545 19843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:15.580837 19843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:15.584313 19843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47"
format_stamp: "Formatted at 2026-09-27 18:43:15 on dist-test-slave-9jll"
server_key: "4a9e13a95e4b8cb7b726ae6de07c727c"
server_key_iv: "ab0a681bc2da69e9e846360569b4a90c"
server_key_version: "encryptionkey@0"
I20260927 18:43:15.585181 19843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47"
format_stamp: "Formatted at 2026-09-27 18:43:15 on dist-test-slave-9jll"
server_key: "4a9e13a95e4b8cb7b726ae6de07c727c"
server_key_iv: "ab0a681bc2da69e9e846360569b4a90c"
server_key_version: "encryptionkey@0"
I20260927 18:43:15.591789 19843 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 18:43:15.596158 19858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:15.597944 19843 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:43:15.598145 19843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47"
format_stamp: "Formatted at 2026-09-27 18:43:15 on dist-test-slave-9jll"
server_key: "4a9e13a95e4b8cb7b726ae6de07c727c"
server_key_iv: "ab0a681bc2da69e9e846360569b4a90c"
server_key_version: "encryptionkey@0"
I20260927 18:43:15.598502 19843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:15.611529 19843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:15.616812 19843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:15.617185 19843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:15.618571 19843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:15.620246 19843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:15.620334 19843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:15.620474 19843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:15.620532 19843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:15.656071 19843 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:46473
I20260927 18:43:15.656116 19970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:46473 every 8 connection(s)
I20260927 18:43:15.657630 19843 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:43:15.665292 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 19843
I20260927 18:43:15.665545 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:43:15.666050 2429 external_mini_cluster.cc:1428] Setting key 60b439837461a69d9d0c8447ca565856
Sep 27 18:43:15 dist-test-slave-9jll krb5kdc[19603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:43:15.681440 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:15.660627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:39807 (local address 127.2.95.126:37217)
0927 18:43:15.661163 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:15.661170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:15.662332 (+ 1162us) server_negotiation.cc:407] Connection header received
0927 18:43:15.664335 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:15.664340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:15.664426 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:15.664541 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:15.666459 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:15.667478 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:15.669478 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:15.670255 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:15.670404 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:15.674368 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:15.674374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:15.674379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:15.674421 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:15.677732 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:15.678555 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:15.678560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:15.678564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:15.678723 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:15.679138 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:15.679142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:15.679145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:15.679531 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:15.679697 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:15.680706 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0927 18:43:15.681091 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":279,"threads_started":1}
I20260927 18:43:15.682945 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:15.661270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37217 (local address 127.2.95.66:39807)
0927 18:43:15.662149 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:15.662191 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:15.664022 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:15.664792 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:15.664802 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:15.665444 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:15.666256 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:15.666280 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:15.667636 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:15.667642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:15.669305 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:15.669321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:15.670500 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:15.670506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:15.670728 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:15.671466 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:15.671494 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:15.674100 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:15.677910 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:15.677918 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:15.677946 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:15.678428 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:15.678814 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:15.678817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:15.678819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:15.679041 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:15.679673 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:15.679683 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:15.680251 (+ 568us) client_negotiation.cc:769] Sending connection context
0927 18:43:15.680598 (+ 347us) client_negotiation.cc:240] Negotiation successful
0927 18:43:15.681295 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":347,"threads_started":1}
I20260927 18:43:15.684741 19971 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37217
I20260927 18:43:15.685163 19971 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:15.686108 19971 heartbeater.cc:511] Master 127.2.95.126:37217 requested a full tablet report, sending...
I20260927 18:43:15.688001 19648 ts_manager.cc:194] Registered new tserver with Master: 7a8c9eeb6dbd4e038f1b597911b1ef47 (127.2.95.66:46473)
I20260927 18:43:15.689065 19648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:39807
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:43:15.734609 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37217
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:43:16.084486 19837 heartbeater.cc:503] Master 127.2.95.126:37217 was elected leader, sending a full tablet report...
W20260927 18:43:16.134757 19977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:16.135146 19977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:16.135281 19977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:16.143277 19977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:16.143452 19977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:16.143621 19977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:43:16.155097 19977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:37217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:16.157073 19977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:16.159228 19977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:16.180279 19982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:16.180358 19985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:16.180358 19983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:16.181936 19977 server_base.cc:1034] running on GCE node
I20260927 18:43:16.182884 19977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:16.184216 19977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:16.185490 19977 hybrid_clock.cc:648] HybridClock initialized: now 1790534596185401 us; error 79 us; skew 500 ppm
I20260927 18:43:16.185916 19977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:16 dist-test-slave-9jll krb5kdc[19603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:16.189947 19977 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:43:16.192086 19977 webserver.cc:460] Webserver started at http://127.2.95.67:37697/ using document root <none> and password file <none>
I20260927 18:43:16.193022 19977 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:16.193157 19977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:16.193517 19977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:16.197114 19977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "b82d997dff9d4f4880d51ac293883a1e"
format_stamp: "Formatted at 2026-09-27 18:43:16 on dist-test-slave-9jll"
server_key: "11cad3eb332edfde6b97261c3a018625"
server_key_iv: "fd966ab0a50ec40167f1e6c5a5214878"
server_key_version: "encryptionkey@0"
I20260927 18:43:16.198098 19977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "b82d997dff9d4f4880d51ac293883a1e"
format_stamp: "Formatted at 2026-09-27 18:43:16 on dist-test-slave-9jll"
server_key: "11cad3eb332edfde6b97261c3a018625"
server_key_iv: "fd966ab0a50ec40167f1e6c5a5214878"
server_key_version: "encryptionkey@0"
I20260927 18:43:16.204370 19977 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:43:16.208700 19992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:16.210783 19977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 18:43:16.211010 19977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "b82d997dff9d4f4880d51ac293883a1e"
format_stamp: "Formatted at 2026-09-27 18:43:16 on dist-test-slave-9jll"
server_key: "11cad3eb332edfde6b97261c3a018625"
server_key_iv: "fd966ab0a50ec40167f1e6c5a5214878"
server_key_version: "encryptionkey@0"
I20260927 18:43:16.211253 19977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:16.230572 19977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:16.235769 19977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:16.236095 19977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:16.237381 19977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:16.239091 19977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:16.239169 19977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:16.239308 19977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:16.239372 19977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:16.275959 19977 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:34383
I20260927 18:43:16.275980 20104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:34383 every 8 connection(s)
I20260927 18:43:16.277566 19977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:43:16.287961 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 19977
I20260927 18:43:16.288185 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:43:16.288887 2429 external_mini_cluster.cc:1428] Setting key 3be0f9c11904f5f441bd0c36102bac0f
Sep 27 18:43:16 dist-test-slave-9jll krb5kdc[19603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:43:16.302572 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.280846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:41351 (local address 127.2.95.126:37217)
0927 18:43:16.281484 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:16.281491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:16.282862 (+ 1371us) server_negotiation.cc:407] Connection header received
0927 18:43:16.285024 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:16.285029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:16.285116 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:16.285236 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:16.287230 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.288117 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.290569 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.291269 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.291443 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.295770 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:16.295775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:16.295778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:16.295825 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:16.298841 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:16.299693 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:16.299697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:16.299700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:16.299845 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:16.300323 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:16.300326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:16.300328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:16.300645 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:16.300813 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:16.301711 (+ 898us) server_negotiation.cc:299] Negotiation successful
0927 18:43:16.302000 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":282,"threads_started":1}
I20260927 18:43:16.304033 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.281579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:37217 (local address 127.2.95.67:41351)
0927 18:43:16.282633 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:16.282688 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:16.284704 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:16.285535 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:16.285545 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:16.286261 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:16.287017 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:16.287037 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.288316 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.288323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:16.290353 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:16.290366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.291581 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.291589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:16.291951 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.292853 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:16.292890 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:16.295500 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:16.299008 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:16.299021 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:16.299045 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:16.299556 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:16.299956 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:16.299961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:16.299965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:16.300191 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:16.300779 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:16.300788 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:16.301231 (+ 443us) client_negotiation.cc:769] Sending connection context
0927 18:43:16.301527 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 18:43:16.302265 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":457,"threads_started":1}
I20260927 18:43:16.306054 20105 heartbeater.cc:347] Connected to a master server at 127.2.95.126:37217
I20260927 18:43:16.306432 20105 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:16.307340 20105 heartbeater.cc:511] Master 127.2.95.126:37217 requested a full tablet report, sending...
I20260927 18:43:16.309194 19648 ts_manager.cc:194] Registered new tserver with Master: b82d997dff9d4f4880d51ac293883a1e (127.2.95.67:34383)
I20260927 18:43:16.310209 19648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:41351
I20260927 18:43:16.319028 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:43:16.346876 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.328765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.2.95.126:37217)
0927 18:43:16.329015 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:16.329023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:16.329414 (+ 391us) server_negotiation.cc:407] Connection header received
0927 18:43:16.329639 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:16.329646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:16.329777 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:16.329882 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:16.332222 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.333477 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.335664 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.336332 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.336483 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.339991 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:16.339999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:16.340005 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:16.340057 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:16.343501 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:16.344319 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:16.344324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:16.344328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:16.344521 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:16.344947 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:16.344952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:16.344956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:16.345503 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:16.345804 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:16.346195 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 18:43:16.346484 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:43:16.352380 19648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:43:16.360410 19648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:43:16.401008 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.376130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59218 (local address 127.2.95.66:46473)
0927 18:43:16.390750 (+ 14620us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:16.390759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:16.390785 (+ 26us) server_negotiation.cc:407] Connection header received
0927 18:43:16.390883 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:16.390890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:16.391100 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:16.391277 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:16.392946 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.394476 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.398346 (+ 3870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.399749 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.399876 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.400183 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:16.400449 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 18:43:16.400649 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14447,"thread_start_us":14331,"threads_started":1}
I20260927 18:43:16.401237 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.376117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:46473 (local address 127.0.0.1:59218)
0927 18:43:16.389521 (+ 13404us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:16.389544 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:16.389708 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:16.391604 (+ 1896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:16.391613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:16.391664 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:16.392684 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:16.392699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.394598 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.394604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:16.397869 (+ 3265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:16.397884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.399951 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.399958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:16.400160 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.400223 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 18:43:16.400381 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 18:43:16.400632 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13233,"mutex_wait_us":388,"spinlock_wait_cycles":896,"thread_start_us":11839,"threads_started":1}
I20260927 18:43:16.401322 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.376118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:33225 (local address 127.0.0.1:42488)
0927 18:43:16.387532 (+ 11414us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:16.387593 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:16.387842 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:16.391604 (+ 3762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:16.391613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:16.391670 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:16.392545 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:16.392568 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.394650 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.394659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:16.397868 (+ 3209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:16.397884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.400339 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.400345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:16.400547 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.400580 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:43:16.400669 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:43:16.400827 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11031,"mutex_wait_us":177,"spinlock_wait_cycles":6912,"thread_start_us":13017,"threads_started":1}
I20260927 18:43:16.402181 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.376605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.2.95.65:33225)
0927 18:43:16.390386 (+ 13781us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:16.390397 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:16.390435 (+ 38us) server_negotiation.cc:407] Connection header received
0927 18:43:16.390585 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:16.390594 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:16.390935 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:16.391176 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:16.392798 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.394541 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.398111 (+ 3570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.399778 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.399913 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.400177 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:16.401558 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0927 18:43:16.401805 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13573,"thread_start_us":13378,"threads_started":1}
I20260927 18:43:16.402263 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.390054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45014 (local address 127.2.95.67:34383)
0927 18:43:16.393577 (+ 3523us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:16.393585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:16.394162 (+ 577us) server_negotiation.cc:407] Connection header received
0927 18:43:16.394308 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:16.394316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:16.394599 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:16.394800 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:16.395608 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.396767 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.398853 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.400309 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.400429 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.400655 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:16.401285 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 18:43:16.401899 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3321,"thread_start_us":1998,"threads_started":1}
I20260927 18:43:16.403111 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.392557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:34383 (local address 127.0.0.1:45014)
0927 18:43:16.394027 (+ 1470us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:16.394046 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:16.394174 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:16.394808 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:16.394812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:16.394843 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:16.395416 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:16.395426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.396909 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.396913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:16.398664 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:16.398678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.400859 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.400866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:16.401063 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.401096 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:43:16.402520 (+ 1424us) client_negotiation.cc:240] Negotiation successful
0927 18:43:16.402730 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"mutex_wait_us":1,"thread_start_us":1196,"threads_started":1}
I20260927 18:43:16.406170 19906 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0f20e01b7841c095994e04a2813856 (DEFAULT_TABLE table=test-table [id=861fa80680f64993b7f789325d6caa1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:16.406523 19772 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0f20e01b7841c095994e04a2813856 (DEFAULT_TABLE table=test-table [id=861fa80680f64993b7f789325d6caa1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:16.407083 20040 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0f20e01b7841c095994e04a2813856 (DEFAULT_TABLE table=test-table [id=861fa80680f64993b7f789325d6caa1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:16.408066 19906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0f20e01b7841c095994e04a2813856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:16.408171 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0f20e01b7841c095994e04a2813856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:16.408629 20040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0f20e01b7841c095994e04a2813856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:16.422441 20124 tablet_bootstrap.cc:492] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47: Bootstrap starting.
I20260927 18:43:16.426532 20125 tablet_bootstrap.cc:492] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d: Bootstrap starting.
I20260927 18:43:16.427495 20126 tablet_bootstrap.cc:492] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e: Bootstrap starting.
I20260927 18:43:16.429351 20124 tablet_bootstrap.cc:654] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:16.430946 20125 tablet_bootstrap.cc:654] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:16.431056 20124 log.cc:824] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:16.431290 20126 tablet_bootstrap.cc:654] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:16.432207 20125 log.cc:824] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:16.432574 20126 log.cc:824] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:16.435035 20124 tablet_bootstrap.cc:492] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47: No bootstrap required, opened a new log
I20260927 18:43:16.435390 20124 ts_tablet_manager.cc:1397] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260927 18:43:16.436002 20126 tablet_bootstrap.cc:492] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e: No bootstrap required, opened a new log
I20260927 18:43:16.436297 20126 ts_tablet_manager.cc:1397] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 18:43:16.440182 20125 tablet_bootstrap.cc:492] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d: No bootstrap required, opened a new log
I20260927 18:43:16.440577 20125 ts_tablet_manager.cc:1397] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260927 18:43:16.447759 20124 raft_consensus.cc:348] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 46473 } } peers { permanent_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33225 } } peers { permanent_uuid: "b82d997dff9d4f4880d51ac293883a1e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 34383 } }
I20260927 18:43:16.447729 20126 raft_consensus.cc:348] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 46473 } } peers { permanent_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33225 } } peers { permanent_uuid: "b82d997dff9d4f4880d51ac293883a1e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 34383 } }
I20260927 18:43:16.448357 20126 raft_consensus.cc:374] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:16.448357 20124 raft_consensus.cc:374] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:16.448483 20124 raft_consensus.cc:729] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8c9eeb6dbd4e038f1b597911b1ef47, State: Initialized, Role: FOLLOWER
I20260927 18:43:16.448506 20126 raft_consensus.cc:729] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b82d997dff9d4f4880d51ac293883a1e, State: Initialized, Role: FOLLOWER
I20260927 18:43:16.448604 20125 raft_consensus.cc:348] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 46473 } } peers { permanent_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33225 } } peers { permanent_uuid: "b82d997dff9d4f4880d51ac293883a1e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 34383 } }
I20260927 18:43:16.449146 20125 raft_consensus.cc:374] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:16.449247 20125 raft_consensus.cc:729] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4bf30acbb8d46e3a67d4ed6196aaf9d, State: Initialized, Role: FOLLOWER
I20260927 18:43:16.449278 20126 consensus_queue.cc:260] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 46473 } } peers { permanent_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33225 } } peers { permanent_uuid: "b82d997dff9d4f4880d51ac293883a1e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 34383 } }
I20260927 18:43:16.449293 20124 consensus_queue.cc:260] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 46473 } } peers { permanent_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33225 } } peers { permanent_uuid: "b82d997dff9d4f4880d51ac293883a1e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 34383 } }
I20260927 18:43:16.449891 20125 consensus_queue.cc:260] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 46473 } } peers { permanent_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33225 } } peers { permanent_uuid: "b82d997dff9d4f4880d51ac293883a1e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 34383 } }
I20260927 18:43:16.450886 19971 heartbeater.cc:503] Master 127.2.95.126:37217 was elected leader, sending a full tablet report...
I20260927 18:43:16.451758 20124 ts_tablet_manager.cc:1428] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260927 18:43:16.452904 20105 heartbeater.cc:503] Master 127.2.95.126:37217 was elected leader, sending a full tablet report...
I20260927 18:43:16.453593 20126 ts_tablet_manager.cc:1428] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e: Time spent starting tablet: real 0.017s user 0.012s sys 0.006s
I20260927 18:43:16.457146 20125 ts_tablet_manager.cc:1428] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
W20260927 18:43:16.529623 20106 tablet.cc:2369] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:43:16.550186 19838 tablet.cc:2369] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:43:16.638104 20131 raft_consensus.cc:482] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:43:16.638547 20131 raft_consensus.cc:504] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 46473 } } peers { permanent_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33225 } } peers { permanent_uuid: "b82d997dff9d4f4880d51ac293883a1e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 34383 } }
I20260927 18:43:16.640949 20131 leader_election.cc:288] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a8c9eeb6dbd4e038f1b597911b1ef47 (127.2.95.66:46473), f4bf30acbb8d46e3a67d4ed6196aaf9d (127.2.95.65:33225)
I20260927 18:43:16.648048 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.641660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:40473 (local address 127.2.95.66:46473)
0927 18:43:16.641938 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:16.641944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:16.641978 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:43:16.642076 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:16.642081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:16.642172 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:16.642269 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:16.642768 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.643886 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.645871 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.647020 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.647149 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.647241 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:16.647649 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 18:43:16.647773 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 18:43:16.648119 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.641309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:46473 (local address 127.2.95.67:40473)
0927 18:43:16.641574 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:16.641604 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:16.641811 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:16.642330 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:16.642335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:16.642369 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:16.642603 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:16.642615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.644041 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.644046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:16.645696 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:16.645714 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.647189 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.647195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:16.647452 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.647503 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 18:43:16.647609 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 18:43:16.647766 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 18:43:16.648835 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.641865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:41781 (local address 127.2.95.65:33225)
0927 18:43:16.642091 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:16.642096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:16.642574 (+ 478us) server_negotiation.cc:407] Connection header received
0927 18:43:16.642707 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:16.642712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:16.642800 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:16.642871 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:16.643548 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.644572 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.646721 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.647870 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.648058 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.648181 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:16.648505 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 18:43:16.648633 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 18:43:16.649485 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.641424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:33225 (local address 127.2.95.67:41781)
0927 18:43:16.642442 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:16.642456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:16.642577 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:16.642928 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:16.642933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:16.642960 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:16.643374 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:16.643388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.644751 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.644755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:16.646556 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:16.646572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.648096 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.648110 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:16.648297 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.648329 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:43:16.649035 (+ 706us) client_negotiation.cc:240] Negotiation successful
0927 18:43:16.649177 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":692,"threads_started":1}
I20260927 18:43:16.649660 19926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0f20e01b7841c095994e04a2813856" candidate_uuid: "b82d997dff9d4f4880d51ac293883a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47" is_pre_election: true
I20260927 18:43:16.650595 19926 raft_consensus.cc:2454] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b82d997dff9d4f4880d51ac293883a1e in term 0.
I20260927 18:43:16.650804 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0f20e01b7841c095994e04a2813856" candidate_uuid: "b82d997dff9d4f4880d51ac293883a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d" is_pre_election: true
I20260927 18:43:16.651369 19792 raft_consensus.cc:2454] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b82d997dff9d4f4880d51ac293883a1e in term 0.
I20260927 18:43:16.651471 19996 leader_election.cc:302] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a8c9eeb6dbd4e038f1b597911b1ef47, b82d997dff9d4f4880d51ac293883a1e; no voters:
I20260927 18:43:16.652000 20131 raft_consensus.cc:2781] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:43:16.652139 20131 raft_consensus.cc:482] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:43:16.652218 20131 raft_consensus.cc:3037] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:16.654803 20131 raft_consensus.cc:504] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 46473 } } peers { permanent_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33225 } } peers { permanent_uuid: "b82d997dff9d4f4880d51ac293883a1e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 34383 } }
I20260927 18:43:16.655723 20131 leader_election.cc:288] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [CANDIDATE]: Term 1 election: Requested vote from peers 7a8c9eeb6dbd4e038f1b597911b1ef47 (127.2.95.66:46473), f4bf30acbb8d46e3a67d4ed6196aaf9d (127.2.95.65:33225)
I20260927 18:43:16.656380 19926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0f20e01b7841c095994e04a2813856" candidate_uuid: "b82d997dff9d4f4880d51ac293883a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47"
I20260927 18:43:16.656374 19792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0f20e01b7841c095994e04a2813856" candidate_uuid: "b82d997dff9d4f4880d51ac293883a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d"
I20260927 18:43:16.656631 19926 raft_consensus.cc:3037] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:16.656682 19792 raft_consensus.cc:3037] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:16.659569 19926 raft_consensus.cc:2454] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b82d997dff9d4f4880d51ac293883a1e in term 1.
W20260927 18:43:16.659976 19972 tablet.cc:2369] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:43:16.660259 19792 raft_consensus.cc:2454] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b82d997dff9d4f4880d51ac293883a1e in term 1.
I20260927 18:43:16.660930 19996 leader_election.cc:302] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a8c9eeb6dbd4e038f1b597911b1ef47, b82d997dff9d4f4880d51ac293883a1e, f4bf30acbb8d46e3a67d4ed6196aaf9d; no voters:
I20260927 18:43:16.661298 20131 raft_consensus.cc:2781] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:16.661919 20131 raft_consensus.cc:686] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 1 LEADER]: Becoming Leader. State: Replica: b82d997dff9d4f4880d51ac293883a1e, State: Running, Role: LEADER
I20260927 18:43:16.662554 20131 consensus_queue.cc:237] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 46473 } } peers { permanent_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33225 } } peers { permanent_uuid: "b82d997dff9d4f4880d51ac293883a1e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 34383 } }
I20260927 18:43:16.668916 19647 catalog_manager.cc:5495] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e reported cstate change: term changed from 0 to 1, leader changed from <none> to b82d997dff9d4f4880d51ac293883a1e (127.2.95.67). New cstate: current_term: 1 leader_uuid: "b82d997dff9d4f4880d51ac293883a1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 46473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b82d997dff9d4f4880d51ac293883a1e" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 34383 } health_report { overall_health: HEALTHY } } }
I20260927 18:43:16.690672 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.684291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.2.95.67:34383)
0927 18:43:16.684586 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:16.684591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:16.684610 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:43:16.684725 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:16.684729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:16.684828 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:16.684985 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:43:16.685446 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.686403 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.687984 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.688424 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.688555 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.689157 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:43:16.689939 (+ 782us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:43:16.690106 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:16.690206 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 18:43:16.690328 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 18:43:16.701918 19926 raft_consensus.cc:1261] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47 [term 1 FOLLOWER]: Refusing update from remote peer b82d997dff9d4f4880d51ac293883a1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:16.701917 19792 raft_consensus.cc:1261] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d [term 1 FOLLOWER]: Refusing update from remote peer b82d997dff9d4f4880d51ac293883a1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:16.703275 20134 consensus_queue.cc:1035] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a8c9eeb6dbd4e038f1b597911b1ef47" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:16.703747 20131 consensus_queue.cc:1035] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4bf30acbb8d46e3a67d4ed6196aaf9d" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:16.725489 20137 mvcc.cc:205] Tried to move back new op lower bound from 7334029708077023232 to 7334029707934298112. Current Snapshot: MvccSnapshot[applied={T|T < 7334029708077023232}]
I20260927 18:43:16.740883 19647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:16.741264 19647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:16.746244 19647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb8ec27a5f994c60a0d629e68b30d20f: Sending DeleteTablet for 3 replicas of tablet 5e0f20e01b7841c095994e04a2813856
I20260927 18:43:16.747890 19906 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0f20e01b7841c095994e04a2813856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:16 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:59218
I20260927 18:43:16.747953 20040 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0f20e01b7841c095994e04a2813856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:16 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:45014
I20260927 18:43:16.747988 19772 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0f20e01b7841c095994e04a2813856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:16 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:42488
I20260927 18:43:16.751220 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 19709
I20260927 18:43:16.759011 20149 tablet_replica.cc:330] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d: stopping tablet replica
I20260927 18:43:16.760504 20149 raft_consensus.cc:2229] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:16.762110 20149 raft_consensus.cc:2258] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:16.764120 20150 tablet_replica.cc:330] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47: stopping tablet replica
I20260927 18:43:16.764420 20148 tablet_replica.cc:330] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e: stopping tablet replica
I20260927 18:43:16.764933 20150 raft_consensus.cc:2229] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:16.764931 20148 raft_consensus.cc:2229] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:43:16.765646 20150 raft_consensus.cc:2258] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:16.765858 20148 raft_consensus.cc:2258] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:16.768536 20150 ts_tablet_manager.cc:1905] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:16.768539 20149 ts_tablet_manager.cc:1905] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 18:43:16 dist-test-slave-9jll krb5kdc[19603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:43:16.772759 20148 ts_tablet_manager.cc:1905] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:16.776826 20150 ts_tablet_manager.cc:1918] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:16.777138 20150 log.cc:1196] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/5e0f20e01b7841c095994e04a2813856
I20260927 18:43:16.777891 20150 ts_tablet_manager.cc:1939] T 5e0f20e01b7841c095994e04a2813856 P 7a8c9eeb6dbd4e038f1b597911b1ef47: Deleting consensus metadata
I20260927 18:43:16.780403 19635 catalog_manager.cc:4841] TS 7a8c9eeb6dbd4e038f1b597911b1ef47 (127.2.95.66:46473): tablet 5e0f20e01b7841c095994e04a2813856 (table test-table [id=861fa80680f64993b7f789325d6caa1a]) successfully deleted
I20260927 18:43:16.784003 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.753264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.2.95.65:33225)
0927 18:43:16.753535 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:16.753544 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:16.759269 (+ 5725us) server_negotiation.cc:407] Connection header received
0927 18:43:16.759647 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:16.759655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:16.759795 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:16.759889 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:16.762123 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.763470 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.765592 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.766316 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.766453 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.773377 (+ 6924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:16.773400 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:16.773423 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:16.773501 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:16.779618 (+ 6117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:16.780744 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:16.780756 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:16.780759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:16.780969 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:16.781504 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:16.781510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:16.781513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:16.782190 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:16.782577 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:16.783238 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 18:43:16.783521 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 18:43:16.784546 20149 ts_tablet_manager.cc:1918] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:16.784801 20149 log.cc:1196] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/5e0f20e01b7841c095994e04a2813856
I20260927 18:43:16.785488 20149 ts_tablet_manager.cc:1939] T 5e0f20e01b7841c095994e04a2813856 P f4bf30acbb8d46e3a67d4ed6196aaf9d: Deleting consensus metadata
I20260927 18:43:16.785972 19832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:16.786370 20148 ts_tablet_manager.cc:1918] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:16.786688 20148 log.cc:1196] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/5e0f20e01b7841c095994e04a2813856
I20260927 18:43:16.787468 20148 ts_tablet_manager.cc:1939] T 5e0f20e01b7841c095994e04a2813856 P b82d997dff9d4f4880d51ac293883a1e: Deleting consensus metadata
I20260927 18:43:16.790301 19634 catalog_manager.cc:4841] TS b82d997dff9d4f4880d51ac293883a1e (127.2.95.67:34383): tablet 5e0f20e01b7841c095994e04a2813856 (table test-table [id=861fa80680f64993b7f789325d6caa1a]) successfully deleted
I20260927 18:43:16.886178 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 19709
I20260927 18:43:16.886250 19634 catalog_manager.cc:4841] TS f4bf30acbb8d46e3a67d4ed6196aaf9d (127.2.95.65:33225): tablet 5e0f20e01b7841c095994e04a2813856 (table test-table [id=861fa80680f64993b7f789325d6caa1a]) successfully deleted
I20260927 18:43:16.913545 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 19843
Sep 27 18:43:16 dist-test-slave-9jll krb5kdc[19603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:43:16.932190 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:16.914727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.2.95.66:46473)
0927 18:43:16.914958 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:16.914965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:16.915002 (+ 37us) server_negotiation.cc:407] Connection header received
0927 18:43:16.915079 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:16.915084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:16.915176 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:16.915265 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:16.916784 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.917645 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.919123 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:16.919575 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:16.919695 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:16.924961 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:16.924985 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:16.925008 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:16.925062 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:16.928684 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:16.929655 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:16.929709 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:16.929714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:16.929889 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:16.930305 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:16.930309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:16.930311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:16.930845 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:16.931175 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:16.931520 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 18:43:16.931801 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:43:16.933074 19966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:17.032086 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 19843
I20260927 18:43:17.049834 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 19977
Sep 27 18:43:17 dist-test-slave-9jll krb5kdc[19603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:43:17.076548 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:17.050717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.2.95.67:34383)
0927 18:43:17.050992 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:17.050998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:17.051248 (+ 250us) server_negotiation.cc:407] Connection header received
0927 18:43:17.051453 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:17.051459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:17.051544 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:17.051654 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:17.053125 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:17.054173 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:17.055565 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:17.056105 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:17.056227 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:17.068964 (+ 12737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:17.068984 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:17.069007 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:17.069064 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:17.072842 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:17.073844 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:17.073859 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:17.073861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:17.074060 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:17.074524 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:17.074528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:17.074530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:17.074997 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:17.075404 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:17.075786 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 18:43:17.076064 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:43:17.077382 20100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:17.216377 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 19977
I20260927 18:43:17.235988 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 19616
I20260927 18:43:17.237552 19678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:17.367193 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 19616
2026-09-27T18:43:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4972 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T18:43:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:17Z Disabled control of system clock
I20260927 18:43:17.457372 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:34417
--webserver_interface=127.2.95.126
--webserver_port=0
--builtin_ntp_servers=127.2.95.84:33635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:34417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:43:17.855118 20159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:17.855532 20159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:17.855651 20159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:17.863641 20159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:17.863802 20159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:17.863842 20159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:17.863919 20159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:17.864001 20159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:17.875567 20159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:34417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:34417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:17.877388 20159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:17.879684 20159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:17.893870 20164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:17.893824 20167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:17.893797 20165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:17.894894 20159 server_base.cc:1034] running on GCE node
I20260927 18:43:17.895959 20159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:17.898066 20159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:17.899304 20159 hybrid_clock.cc:648] HybridClock initialized: now 1790534597899254 us; error 48 us; skew 500 ppm
I20260927 18:43:17.899724 20159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:43:17.902673 20159 webserver.cc:460] Webserver started at http://127.2.95.126:46529/ using document root <none> and password file <none>
I20260927 18:43:17.903612 20159 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:17.903753 20159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:17.904139 20159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:17.908211 20159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "3090c1590bed4caaa000862d96d5e8e1"
format_stamp: "Formatted at 2026-09-27 18:43:17 on dist-test-slave-9jll"
server_key: "ca9f7980a2aa308e0814a18ee8070753"
server_key_iv: "ced91f3a8883ec7f2d9aee21e6e9306f"
server_key_version: "encryptionkey@0"
I20260927 18:43:17.909366 20159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "3090c1590bed4caaa000862d96d5e8e1"
format_stamp: "Formatted at 2026-09-27 18:43:17 on dist-test-slave-9jll"
server_key: "ca9f7980a2aa308e0814a18ee8070753"
server_key_iv: "ced91f3a8883ec7f2d9aee21e6e9306f"
server_key_version: "encryptionkey@0"
I20260927 18:43:17.916586 20159 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 18:43:17.921340 20173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:17.923372 20159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 18:43:17.923593 20159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "3090c1590bed4caaa000862d96d5e8e1"
format_stamp: "Formatted at 2026-09-27 18:43:17 on dist-test-slave-9jll"
server_key: "ca9f7980a2aa308e0814a18ee8070753"
server_key_iv: "ced91f3a8883ec7f2d9aee21e6e9306f"
server_key_version: "encryptionkey@0"
I20260927 18:43:17.923863 20159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:17.946545 20159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:17.947680 20159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:17.948014 20159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:17.968988 20159 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:34417
I20260927 18:43:17.969024 20224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:34417 every 8 connection(s)
I20260927 18:43:17.970719 20159 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:17.975621 20225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:17.979228 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 20159
I20260927 18:43:17.979419 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:17.979960 2429 external_mini_cluster.cc:1428] Setting key e0b553aa88801aa4223e8ba4c22d2d79
I20260927 18:43:17.990370 20225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1: Bootstrap starting.
I20260927 18:43:17.994526 20225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:17.995946 20225 log.cc:824] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:17.996018 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:17.982516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44302 (local address 127.2.95.126:34417)
0927 18:43:17.983728 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:17.983742 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:17.983793 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:43:17.985808 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:17.985838 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:17.986486 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:17.986967 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:17.987696 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:17.989131 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:17.990723 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:17.991504 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:17.991686 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:17.993186 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:17.993195 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:17.993211 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:17.993932 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:17.994046 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:17.994070 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:17.994178 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:17.994439 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 18:43:17.994843 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":419,"threads_started":1}
I20260927 18:43:17.999626 20225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1: No bootstrap required, opened a new log
I20260927 18:43:18.008082 20225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3090c1590bed4caaa000862d96d5e8e1" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 34417 } }
I20260927 18:43:18.008656 20225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:18.008769 20225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3090c1590bed4caaa000862d96d5e8e1, State: Initialized, Role: FOLLOWER
I20260927 18:43:18.009516 20225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3090c1590bed4caaa000862d96d5e8e1" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 34417 } }
I20260927 18:43:18.009838 20225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:18.009991 20225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:18.010169 20225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:18.013053 20225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3090c1590bed4caaa000862d96d5e8e1" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 34417 } }
I20260927 18:43:18.013783 20225 leader_election.cc:302] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3090c1590bed4caaa000862d96d5e8e1; no voters:
I20260927 18:43:18.014534 20225 leader_election.cc:288] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:18.014701 20230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:18.015473 20230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3090c1590bed4caaa000862d96d5e8e1, State: Running, Role: LEADER
I20260927 18:43:18.016093 20230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3090c1590bed4caaa000862d96d5e8e1" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 34417 } }
I20260927 18:43:18.016990 20225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:18.019695 20231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3090c1590bed4caaa000862d96d5e8e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3090c1590bed4caaa000862d96d5e8e1" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 34417 } } }
I20260927 18:43:18.019640 20232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3090c1590bed4caaa000862d96d5e8e1. Latest consensus state: current_term: 1 leader_uuid: "3090c1590bed4caaa000862d96d5e8e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3090c1590bed4caaa000862d96d5e8e1" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 34417 } } }
I20260927 18:43:18.020185 20231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:18.020264 20232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:18.024264 20238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:18.029984 20238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:18.040314 20238 catalog_manager.cc:1347] Generated new cluster ID: e350be82121542c6b5189d57317af181
I20260927 18:43:18.040472 20238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:18.063798 20238 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:18.065577 20238 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:18.075030 20238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1: Generated new TSK 0
I20260927 18:43:18.076175 20238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:18.081862 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:34417
--builtin_ntp_servers=127.2.95.84:33635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:43:18.495349 20249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:18.495745 20249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:18.495885 20249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:18.504289 20249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:18.504455 20249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:18.504626 20249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:43:18.514994 20249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:34417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:18.516750 20249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:18.519197 20249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:18.534590 20254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:18.534590 20257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:18.534636 20255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:18.535585 20249 server_base.cc:1034] running on GCE node
I20260927 18:43:18.536392 20249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:18.537627 20249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:18.538873 20249 hybrid_clock.cc:648] HybridClock initialized: now 1790534598538813 us; error 48 us; skew 500 ppm
I20260927 18:43:18.539251 20249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:43:18.541924 20249 webserver.cc:460] Webserver started at http://127.2.95.65:40773/ using document root <none> and password file <none>
I20260927 18:43:18.542763 20249 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:18.542853 20249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:18.543195 20249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:18.546877 20249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "10115c8fe9f343f2b81ff466eee07d79"
format_stamp: "Formatted at 2026-09-27 18:43:18 on dist-test-slave-9jll"
server_key: "e5bd8c5d0395967a9b3242f4a80429de"
server_key_iv: "81e428a6f53fcef67f6d82d1ed643c93"
server_key_version: "encryptionkey@0"
I20260927 18:43:18.547793 20249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "10115c8fe9f343f2b81ff466eee07d79"
format_stamp: "Formatted at 2026-09-27 18:43:18 on dist-test-slave-9jll"
server_key: "e5bd8c5d0395967a9b3242f4a80429de"
server_key_iv: "81e428a6f53fcef67f6d82d1ed643c93"
server_key_version: "encryptionkey@0"
I20260927 18:43:18.554374 20249 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 18:43:18.558667 20263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:18.560298 20249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:43:18.560497 20249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "10115c8fe9f343f2b81ff466eee07d79"
format_stamp: "Formatted at 2026-09-27 18:43:18 on dist-test-slave-9jll"
server_key: "e5bd8c5d0395967a9b3242f4a80429de"
server_key_iv: "81e428a6f53fcef67f6d82d1ed643c93"
server_key_version: "encryptionkey@0"
I20260927 18:43:18.560737 20249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:18.579631 20249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:18.580679 20249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:18.580977 20249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:18.582154 20249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:18.583940 20249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:18.584040 20249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:18.584156 20249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:18.584232 20249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:18.620188 20249 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:45445
I20260927 18:43:18.620213 20375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:45445 every 8 connection(s)
I20260927 18:43:18.621881 20249 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:43:18.624745 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 20249
I20260927 18:43:18.624953 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:43:18.625566 2429 external_mini_cluster.cc:1428] Setting key cf97a67729bfbc50b11868de822e03f4
I20260927 18:43:18.629310 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:34417
--builtin_ntp_servers=127.2.95.84:33635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:43:18.639412 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:18.624957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:39805 (local address 127.2.95.126:34417)
0927 18:43:18.625615 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:18.625622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:18.627168 (+ 1546us) server_negotiation.cc:407] Connection header received
0927 18:43:18.629893 (+ 2725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:18.629898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:18.629981 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:18.630090 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:18.631885 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:18.633176 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:18.635449 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:18.636128 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:18.636306 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:18.638366 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:18.638375 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:18.638381 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:18.638414 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:18.638476 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:18.638495 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:18.638590 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:18.638926 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 18:43:18.639068 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":337,"threads_started":1}
I20260927 18:43:18.642143 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:18.625787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:34417 (local address 127.2.95.65:39805)
0927 18:43:18.627001 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:18.627044 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:18.629500 (+ 2456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:18.630419 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:18.630446 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:18.630606 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:18.631648 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:18.631673 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:18.633369 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:18.633375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:18.635255 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:18.635270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:18.636471 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:18.636482 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:18.636860 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:18.637854 (+ 994us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:18.637858 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:18.637926 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:18.638060 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:18.638608 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:18.638621 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:18.638632 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 18:43:18.638836 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 18:43:18.640491 (+ 1655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":348,"threads_started":1}
I20260927 18:43:18.644610 20376 heartbeater.cc:347] Connected to a master server at 127.2.95.126:34417
I20260927 18:43:18.645051 20376 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:18.646219 20376 heartbeater.cc:511] Master 127.2.95.126:34417 requested a full tablet report, sending...
I20260927 18:43:18.649413 20190 ts_manager.cc:194] Registered new tserver with Master: 10115c8fe9f343f2b81ff466eee07d79 (127.2.95.65:45445)
I20260927 18:43:18.651702 20190 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.95.65:39805
W20260927 18:43:19.028893 20380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:19.029258 20380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:19.029356 20380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:19.037427 20380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:19.037564 20380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:19.037719 20380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:43:19.047894 20380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:34417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:19.049896 20380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:19.052583 20380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:19.069120 20386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:19.069203 20388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:19.069234 20385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:19.070089 20380 server_base.cc:1034] running on GCE node
I20260927 18:43:19.070897 20380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:19.072170 20380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:19.073449 20380 hybrid_clock.cc:648] HybridClock initialized: now 1790534599073375 us; error 49 us; skew 500 ppm
I20260927 18:43:19.073911 20380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:43:19.077128 20380 webserver.cc:460] Webserver started at http://127.2.95.66:43319/ using document root <none> and password file <none>
I20260927 18:43:19.078064 20380 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:19.078166 20380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:19.078516 20380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:19.082556 20380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "11a7e1d48ca947b68c1328540f0a8937"
format_stamp: "Formatted at 2026-09-27 18:43:19 on dist-test-slave-9jll"
server_key: "c3784ce9694a1950eb319336495cd023"
server_key_iv: "d87a562a692b87759a56e7267625f2ed"
server_key_version: "encryptionkey@0"
I20260927 18:43:19.083482 20380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "11a7e1d48ca947b68c1328540f0a8937"
format_stamp: "Formatted at 2026-09-27 18:43:19 on dist-test-slave-9jll"
server_key: "c3784ce9694a1950eb319336495cd023"
server_key_iv: "d87a562a692b87759a56e7267625f2ed"
server_key_version: "encryptionkey@0"
I20260927 18:43:19.090384 20380 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260927 18:43:19.095177 20394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:19.097146 20380 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260927 18:43:19.097352 20380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "11a7e1d48ca947b68c1328540f0a8937"
format_stamp: "Formatted at 2026-09-27 18:43:19 on dist-test-slave-9jll"
server_key: "c3784ce9694a1950eb319336495cd023"
server_key_iv: "d87a562a692b87759a56e7267625f2ed"
server_key_version: "encryptionkey@0"
I20260927 18:43:19.097657 20380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:19.117333 20380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:19.118535 20380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:19.118836 20380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:19.120024 20380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:19.121902 20380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:19.122026 20380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:19.122151 20380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:19.122229 20380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:19.157131 20380 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:39157
I20260927 18:43:19.157203 20506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:39157 every 8 connection(s)
I20260927 18:43:19.158768 20380 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:43:19.169181 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 20380
I20260927 18:43:19.169361 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:43:19.169927 2429 external_mini_cluster.cc:1428] Setting key e95266c34360337ac11bb91c6376fa09
I20260927 18:43:19.173270 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:34417
--builtin_ntp_servers=127.2.95.84:33635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:43:19.174569 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:19.161816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:41193 (local address 127.2.95.126:34417)
0927 18:43:19.162331 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:19.162337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:19.163892 (+ 1555us) server_negotiation.cc:407] Connection header received
0927 18:43:19.166153 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:19.166160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:19.166260 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:19.166512 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:19.167983 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.168927 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.170930 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.171462 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.171568 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:19.173441 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:19.173450 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:19.173455 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:19.173486 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:19.173547 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:19.173567 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:19.173652 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:19.173961 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 18:43:19.174157 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":248,"threads_started":1}
I20260927 18:43:19.175779 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:19.162416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:34417 (local address 127.2.95.66:41193)
0927 18:43:19.163685 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:19.163727 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:19.165788 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:19.166824 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:19.166835 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:19.167012 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:19.167762 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:19.167782 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.169133 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.169138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:19.170778 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:19.170794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.171733 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.171741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:19.172053 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:19.172849 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:19.172852 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:19.172908 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:19.173033 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:19.173720 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:19.173732 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:19.173742 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 18:43:19.173931 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 18:43:19.174431 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":405,"threads_started":1}
I20260927 18:43:19.177522 20507 heartbeater.cc:347] Connected to a master server at 127.2.95.126:34417
I20260927 18:43:19.177954 20507 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:19.179068 20507 heartbeater.cc:511] Master 127.2.95.126:34417 requested a full tablet report, sending...
I20260927 18:43:19.181054 20190 ts_manager.cc:194] Registered new tserver with Master: 11a7e1d48ca947b68c1328540f0a8937 (127.2.95.66:39157)
I20260927 18:43:19.182127 20190 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.95.66:41193
W20260927 18:43:19.572877 20511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:19.573246 20511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:19.573370 20511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:19.581970 20511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:19.582122 20511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:19.582288 20511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:43:19.593109 20511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:34417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:19.594935 20511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:19.597268 20511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:19.614008 20516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:19.613921 20517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:19.614217 20519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:19.614849 20511 server_base.cc:1034] running on GCE node
I20260927 18:43:19.615639 20511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:19.616855 20511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:19.618101 20511 hybrid_clock.cc:648] HybridClock initialized: now 1790534599618039 us; error 51 us; skew 500 ppm
I20260927 18:43:19.618547 20511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:43:19.621706 20511 webserver.cc:460] Webserver started at http://127.2.95.67:37711/ using document root <none> and password file <none>
I20260927 18:43:19.622717 20511 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:19.622839 20511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:19.623193 20511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:19.627009 20511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "4f55629a23c34d8d9a8bc530b5151b58"
format_stamp: "Formatted at 2026-09-27 18:43:19 on dist-test-slave-9jll"
server_key: "914d4443ed230d2be136ff9f7da23051"
server_key_iv: "9880a924f1d0b3636646c6d135ed4c76"
server_key_version: "encryptionkey@0"
I20260927 18:43:19.627970 20511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "4f55629a23c34d8d9a8bc530b5151b58"
format_stamp: "Formatted at 2026-09-27 18:43:19 on dist-test-slave-9jll"
server_key: "914d4443ed230d2be136ff9f7da23051"
server_key_iv: "9880a924f1d0b3636646c6d135ed4c76"
server_key_version: "encryptionkey@0"
I20260927 18:43:19.634053 20511 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 18:43:19.638196 20525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:19.639806 20511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:43:19.639992 20511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "4f55629a23c34d8d9a8bc530b5151b58"
format_stamp: "Formatted at 2026-09-27 18:43:19 on dist-test-slave-9jll"
server_key: "914d4443ed230d2be136ff9f7da23051"
server_key_iv: "9880a924f1d0b3636646c6d135ed4c76"
server_key_version: "encryptionkey@0"
I20260927 18:43:19.640242 20511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:19.655910 20376 heartbeater.cc:503] Master 127.2.95.126:34417 was elected leader, sending a full tablet report...
I20260927 18:43:19.664361 20511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:19.665478 20511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:19.665822 20511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:19.667018 20511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:19.668875 20511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:19.668982 20511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:19.669100 20511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:19.669174 20511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:19.705163 20511 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:38387
I20260927 18:43:19.705217 20637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:38387 every 8 connection(s)
I20260927 18:43:19.706849 20511 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:43:19.717222 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 20511
I20260927 18:43:19.717397 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:43:19.717962 2429 external_mini_cluster.cc:1428] Setting key bb676e69c7092701cb1cd5b557881a7b
I20260927 18:43:19.722136 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:19.710010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:56729 (local address 127.2.95.126:34417)
0927 18:43:19.710637 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:19.710644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:19.711947 (+ 1303us) server_negotiation.cc:407] Connection header received
0927 18:43:19.714135 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:19.714140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:19.714236 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:19.714351 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:19.715698 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.716553 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.718611 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.719102 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.719232 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:19.721037 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:19.721043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:19.721048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:19.721071 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:19.721127 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:19.721146 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:19.721247 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:19.721621 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 18:43:19.721762 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":271,"threads_started":1}
I20260927 18:43:19.723299 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:19.710542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:34417 (local address 127.2.95.67:56729)
0927 18:43:19.711668 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:19.711718 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:19.713810 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:19.714618 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:19.714631 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:19.714734 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:19.715513 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:19.715534 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.716694 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.716698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:19.718455 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:19.718472 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.719340 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.719350 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:19.719675 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:19.720674 (+ 999us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:19.720677 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:19.720736 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:19.720840 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:19.721278 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:19.721288 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:19.721296 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:43:19.721540 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 18:43:19.722060 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":341,"threads_started":1}
I20260927 18:43:19.725036 20638 heartbeater.cc:347] Connected to a master server at 127.2.95.126:34417
I20260927 18:43:19.725390 20638 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:19.726365 20638 heartbeater.cc:511] Master 127.2.95.126:34417 requested a full tablet report, sending...
I20260927 18:43:19.728339 20190 ts_manager.cc:194] Registered new tserver with Master: 4f55629a23c34d8d9a8bc530b5151b58 (127.2.95.67:38387)
I20260927 18:43:19.729295 20190 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.95.67:56729
I20260927 18:43:19.733981 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:43:19.745085 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:19.738558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.2.95.126:34417)
0927 18:43:19.738825 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:19.738831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:19.739147 (+ 316us) server_negotiation.cc:407] Connection header received
0927 18:43:19.739331 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:19.739335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:19.739422 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:19.739531 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:19.740125 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.740974 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.742613 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.743095 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.743207 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:19.744350 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:19.744355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:19.744359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:19.744380 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:19.744419 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:19.744432 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:19.744512 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:19.744628 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 18:43:19.744728 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 18:43:19.750274 20190 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:43:19.758635 20190 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:43:19.786851 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:19.776934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.2.95.66:39157)
0927 18:43:19.778497 (+ 1563us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:19.778504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:19.778532 (+ 28us) server_negotiation.cc:407] Connection header received
0927 18:43:19.778641 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:19.778648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:19.778853 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:19.779043 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:19.780238 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.781396 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.784367 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.785591 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.785753 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:19.786069 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:19.786278 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 18:43:19.786487 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1408,"thread_start_us":1260,"threads_started":1}
I20260927 18:43:19.786927 20655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:19.776626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:39157 (local address 127.0.0.1:53698)
0927 18:43:19.777902 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:19.777917 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:19.778011 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:19.779281 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:19.779285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:19.779416 (+ 131us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:19.779987 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:19.780014 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.781542 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.781547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:19.784142 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:19.784157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.785805 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.785811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:19.786020 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:19.786072 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 18:43:19.786207 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 18:43:19.786440 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"mutex_wait_us":27,"spinlock_wait_cycles":8448,"thread_start_us":1028,"threads_started":1}
I20260927 18:43:19.787719 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:19.775008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:45445 (local address 127.0.0.1:35910)
0927 18:43:19.777617 (+ 2609us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:19.777636 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:19.777936 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:19.779107 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:19.779116 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:19.779149 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:19.780084 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:19.780097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.782262 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.782269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:19.784597 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:19.784614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.786930 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.786937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:19.787109 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:19.787140 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:43:19.787229 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:43:19.787362 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2466,"mutex_wait_us":508,"thread_start_us":2302,"threads_started":1}
I20260927 18:43:19.788177 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:19.775839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.2.95.65:45445)
0927 18:43:19.777126 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:19.777137 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:19.777805 (+ 668us) server_negotiation.cc:407] Connection header received
0927 18:43:19.778178 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:19.778189 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:19.778528 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:19.778741 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:19.780268 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.782089 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.784826 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.786325 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.786460 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:19.786729 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:19.787566 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 18:43:19.787774 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":914,"threads_started":1}
I20260927 18:43:19.789155 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:19.774733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:38387 (local address 127.0.0.1:41226)
0927 18:43:19.775650 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:19.775693 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:19.775887 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:19.779474 (+ 3587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:19.779478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:19.779518 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:19.780148 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:19.780162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.782221 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.782227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:19.784643 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:19.784653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.786878 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.786884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:19.788528 (+ 1644us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:19.788562 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:43:19.788652 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 18:43:19.788796 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":249,"spinlock_wait_cycles":896,"thread_start_us":423,"threads_started":1}
I20260927 18:43:19.789938 20653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:19.775039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.2.95.67:38387)
0927 18:43:19.778316 (+ 3277us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:19.778324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:19.778488 (+ 164us) server_negotiation.cc:407] Connection header received
0927 18:43:19.778606 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:19.778613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:19.778860 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:19.779067 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:19.780382 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.782072 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.784850 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:19.786325 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:19.786467 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:19.786729 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:19.789376 (+ 2647us) server_negotiation.cc:299] Negotiation successful
0927 18:43:19.789582 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3078,"thread_start_us":1353,"threads_started":1}
I20260927 18:43:19.792193 20311 tablet_service.cc:1467] Processing CreateTablet for tablet 8bdd8431e72a40c4ac2e414f4873537e (DEFAULT_TABLE table=test-table [id=6e045b8cf3644da09a5fbe67f48518a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:19.792193 20442 tablet_service.cc:1467] Processing CreateTablet for tablet 8bdd8431e72a40c4ac2e414f4873537e (DEFAULT_TABLE table=test-table [id=6e045b8cf3644da09a5fbe67f48518a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:19.793013 20573 tablet_service.cc:1467] Processing CreateTablet for tablet 8bdd8431e72a40c4ac2e414f4873537e (DEFAULT_TABLE table=test-table [id=6e045b8cf3644da09a5fbe67f48518a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:19.794049 20311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bdd8431e72a40c4ac2e414f4873537e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:19.794055 20442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bdd8431e72a40c4ac2e414f4873537e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:19.794525 20573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bdd8431e72a40c4ac2e414f4873537e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:19.811014 20657 tablet_bootstrap.cc:492] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58: Bootstrap starting.
I20260927 18:43:19.814937 20659 tablet_bootstrap.cc:492] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937: Bootstrap starting.
I20260927 18:43:19.816020 20657 tablet_bootstrap.cc:654] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:19.817706 20657 log.cc:824] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:19.819319 20658 tablet_bootstrap.cc:492] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79: Bootstrap starting.
I20260927 18:43:19.819355 20659 tablet_bootstrap.cc:654] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:19.820950 20659 log.cc:824] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:19.821594 20657 tablet_bootstrap.cc:492] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58: No bootstrap required, opened a new log
I20260927 18:43:19.821982 20657 ts_tablet_manager.cc:1397] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260927 18:43:19.824054 20658 tablet_bootstrap.cc:654] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:19.825613 20658 log.cc:824] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:19.829257 20659 tablet_bootstrap.cc:492] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937: No bootstrap required, opened a new log
I20260927 18:43:19.829288 20658 tablet_bootstrap.cc:492] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79: No bootstrap required, opened a new log
I20260927 18:43:19.829662 20659 ts_tablet_manager.cc:1397] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260927 18:43:19.829725 20658 ts_tablet_manager.cc:1397] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260927 18:43:19.832204 20657 raft_consensus.cc:348] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 } } peers { permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 } } peers { permanent_uuid: "11a7e1d48ca947b68c1328540f0a8937" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39157 } }
I20260927 18:43:19.832782 20657 raft_consensus.cc:374] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:19.832890 20657 raft_consensus.cc:729] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f55629a23c34d8d9a8bc530b5151b58, State: Initialized, Role: FOLLOWER
I20260927 18:43:19.833694 20657 consensus_queue.cc:260] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 } } peers { permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 } } peers { permanent_uuid: "11a7e1d48ca947b68c1328540f0a8937" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39157 } }
I20260927 18:43:19.835172 20638 heartbeater.cc:503] Master 127.2.95.126:34417 was elected leader, sending a full tablet report...
I20260927 18:43:19.836035 20657 ts_tablet_manager.cc:1428] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 18:43:19.839150 20659 raft_consensus.cc:348] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 } } peers { permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 } } peers { permanent_uuid: "11a7e1d48ca947b68c1328540f0a8937" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39157 } }
I20260927 18:43:19.840090 20659 raft_consensus.cc:374] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:19.840258 20659 raft_consensus.cc:729] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a7e1d48ca947b68c1328540f0a8937, State: Initialized, Role: FOLLOWER
I20260927 18:43:19.840037 20658 raft_consensus.cc:348] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 } } peers { permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 } } peers { permanent_uuid: "11a7e1d48ca947b68c1328540f0a8937" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39157 } }
I20260927 18:43:19.840523 20658 raft_consensus.cc:374] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:19.840638 20658 raft_consensus.cc:729] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10115c8fe9f343f2b81ff466eee07d79, State: Initialized, Role: FOLLOWER
I20260927 18:43:19.841151 20659 consensus_queue.cc:260] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 } } peers { permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 } } peers { permanent_uuid: "11a7e1d48ca947b68c1328540f0a8937" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39157 } }
I20260927 18:43:19.841272 20658 consensus_queue.cc:260] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 } } peers { permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 } } peers { permanent_uuid: "11a7e1d48ca947b68c1328540f0a8937" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39157 } }
I20260927 18:43:19.842487 20507 heartbeater.cc:503] Master 127.2.95.126:34417 was elected leader, sending a full tablet report...
I20260927 18:43:19.843317 20659 ts_tablet_manager.cc:1428] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260927 18:43:19.845561 20658 ts_tablet_manager.cc:1428] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260927 18:43:19.874861 20377 tablet.cc:2369] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:43:19.911284 20508 tablet.cc:2369] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:43:19.958808 20639 tablet.cc:2369] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:43:20.020238 20665 raft_consensus.cc:482] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:43:20.020747 20665 raft_consensus.cc:504] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 } } peers { permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 } } peers { permanent_uuid: "11a7e1d48ca947b68c1328540f0a8937" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39157 } }
I20260927 18:43:20.020977 20664 raft_consensus.cc:482] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:43:20.021385 20664 raft_consensus.cc:504] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 } } peers { permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 } } peers { permanent_uuid: "11a7e1d48ca947b68c1328540f0a8937" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39157 } }
I20260927 18:43:20.023262 20664 leader_election.cc:288] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f55629a23c34d8d9a8bc530b5151b58 (127.2.95.67:38387), 10115c8fe9f343f2b81ff466eee07d79 (127.2.95.65:45445)
I20260927 18:43:20.029142 20665 leader_election.cc:288] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f55629a23c34d8d9a8bc530b5151b58 (127.2.95.67:38387), 11a7e1d48ca947b68c1328540f0a8937 (127.2.95.66:39157)
I20260927 18:43:20.044013 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:20.035949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:54591 (local address 127.2.95.66:39157)
0927 18:43:20.036122 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:20.036127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:20.036143 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:43:20.036197 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:20.036201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:20.036304 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:20.036385 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:20.037285 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.038488 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.041308 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.042534 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.042653 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:20.042765 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:20.043514 (+ 749us) server_negotiation.cc:299] Negotiation successful
0927 18:43:20.043672 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 18:43:20.044122 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:20.035119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:56201 (local address 127.2.95.65:45445)
0927 18:43:20.035357 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:20.035365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:20.035967 (+ 602us) server_negotiation.cc:407] Connection header received
0927 18:43:20.036172 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:20.036180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:20.036311 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:20.036406 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:20.037798 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.038914 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.041768 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.042967 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.043161 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:20.043299 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:20.043665 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 18:43:20.043809 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":66}
I20260927 18:43:20.044421 20653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:20.024432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:36391 (local address 127.2.95.67:38387)
0927 18:43:20.024690 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:20.024698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:20.034635 (+ 9937us) server_negotiation.cc:407] Connection header received
0927 18:43:20.036305 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:20.036311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:20.036401 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:20.036474 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:20.037736 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.038822 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.041696 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.043260 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.043378 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:20.043485 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:20.044024 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 18:43:20.044155 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":85}
I20260927 18:43:20.044570 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:20.023678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:38387 (local address 127.2.95.65:43915)
0927 18:43:20.036590 (+ 12912us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:20.036608 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:20.036807 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:20.037368 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:20.037373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:20.037411 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:20.037798 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:20.037809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.039519 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.039524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:20.041608 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:20.041622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.043839 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.043844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:20.044013 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:20.044052 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 18:43:20.044134 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:43:20.044271 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12717,"thread_start_us":13292,"threads_started":1}
I20260927 18:43:20.044867 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:20.023888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:45445 (local address 127.2.95.66:56201)
0927 18:43:20.035859 (+ 11971us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:20.035876 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:20.036027 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:20.036607 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:20.036612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:20.036641 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:20.037151 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:20.037167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.039094 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.039102 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:20.041509 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:20.041528 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.043304 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.043311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:20.043481 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:20.043516 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:43:20.044375 (+ 859us) client_negotiation.cc:240] Negotiation successful
0927 18:43:20.044520 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11840,"mutex_wait_us":11,"thread_start_us":11661,"threads_started":1}
I20260927 18:43:20.045312 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:20.024802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:43915 (local address 127.2.95.67:38387)
0927 18:43:20.033182 (+ 8380us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:20.033192 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:20.036688 (+ 3496us) server_negotiation.cc:407] Connection header received
0927 18:43:20.036957 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:20.036962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:20.037028 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:20.037101 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:20.038043 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.039323 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.041817 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.043514 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.043627 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:20.043719 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:20.044795 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0927 18:43:20.044938 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":231,"server-negotiator.queue_time_us":8177,"thread_start_us":7862,"threads_started":1}
I20260927 18:43:20.045809 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:20.023591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:39157 (local address 127.2.95.65:54591)
0927 18:43:20.035095 (+ 11504us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:20.035122 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:20.035394 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:20.036592 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:20.036598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:20.036627 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:20.037080 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:20.037097 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.038642 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.038647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:20.041073 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:20.041087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.042895 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.042902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:20.043326 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:20.043359 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:43:20.044315 (+ 956us) client_negotiation.cc:240] Negotiation successful
0927 18:43:20.044475 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11349,"mutex_wait_us":91,"thread_start_us":11083,"threads_started":1}
I20260927 18:43:20.045886 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:20.023659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:38387 (local address 127.2.95.66:36391)
0927 18:43:20.034448 (+ 10789us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:20.034477 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:20.036096 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:20.036652 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:20.036656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:20.036678 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:20.037573 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:20.037586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.038999 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.039004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:20.041455 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:20.041467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.043599 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.043605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:20.043776 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:20.043809 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:43:20.044904 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0927 18:43:20.045187 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10600,"mutex_wait_us":158,"thread_start_us":10381,"threads_started":1}
I20260927 18:43:20.046159 20331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdd8431e72a40c4ac2e414f4873537e" candidate_uuid: "11a7e1d48ca947b68c1328540f0a8937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10115c8fe9f343f2b81ff466eee07d79" is_pre_election: true
I20260927 18:43:20.046442 20593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdd8431e72a40c4ac2e414f4873537e" candidate_uuid: "10115c8fe9f343f2b81ff466eee07d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f55629a23c34d8d9a8bc530b5151b58" is_pre_election: true
I20260927 18:43:20.046772 20331 raft_consensus.cc:2454] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11a7e1d48ca947b68c1328540f0a8937 in term 0.
I20260927 18:43:20.046980 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdd8431e72a40c4ac2e414f4873537e" candidate_uuid: "11a7e1d48ca947b68c1328540f0a8937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f55629a23c34d8d9a8bc530b5151b58" is_pre_election: true
I20260927 18:43:20.047005 20462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdd8431e72a40c4ac2e414f4873537e" candidate_uuid: "10115c8fe9f343f2b81ff466eee07d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11a7e1d48ca947b68c1328540f0a8937" is_pre_election: true
I20260927 18:43:20.047227 20593 raft_consensus.cc:2454] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10115c8fe9f343f2b81ff466eee07d79 in term 0.
I20260927 18:43:20.047458 20462 raft_consensus.cc:2454] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10115c8fe9f343f2b81ff466eee07d79 in term 0.
I20260927 18:43:20.047574 20395 leader_election.cc:302] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10115c8fe9f343f2b81ff466eee07d79, 11a7e1d48ca947b68c1328540f0a8937; no voters:
I20260927 18:43:20.048064 20664 raft_consensus.cc:2781] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:43:20.048053 20267 leader_election.cc:302] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10115c8fe9f343f2b81ff466eee07d79, 4f55629a23c34d8d9a8bc530b5151b58; no voters:
I20260927 18:43:20.048243 20664 raft_consensus.cc:482] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:43:20.048404 20664 raft_consensus.cc:3037] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:20.048630 20665 raft_consensus.cc:2781] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:43:20.048806 20665 raft_consensus.cc:482] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:43:20.048915 20665 raft_consensus.cc:3037] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:20.051083 20664 raft_consensus.cc:504] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 } } peers { permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 } } peers { permanent_uuid: "11a7e1d48ca947b68c1328540f0a8937" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39157 } }
I20260927 18:43:20.051559 20665 raft_consensus.cc:504] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 } } peers { permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 } } peers { permanent_uuid: "11a7e1d48ca947b68c1328540f0a8937" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39157 } }
I20260927 18:43:20.051921 20664 leader_election.cc:288] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [CANDIDATE]: Term 1 election: Requested vote from peers 4f55629a23c34d8d9a8bc530b5151b58 (127.2.95.67:38387), 10115c8fe9f343f2b81ff466eee07d79 (127.2.95.65:45445)
I20260927 18:43:20.052446 20665 leader_election.cc:288] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [CANDIDATE]: Term 1 election: Requested vote from peers 4f55629a23c34d8d9a8bc530b5151b58 (127.2.95.67:38387), 11a7e1d48ca947b68c1328540f0a8937 (127.2.95.66:39157)
I20260927 18:43:20.052515 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdd8431e72a40c4ac2e414f4873537e" candidate_uuid: "11a7e1d48ca947b68c1328540f0a8937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f55629a23c34d8d9a8bc530b5151b58"
I20260927 18:43:20.052789 20592 raft_consensus.cc:3037] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:20.052837 20593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdd8431e72a40c4ac2e414f4873537e" candidate_uuid: "10115c8fe9f343f2b81ff466eee07d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f55629a23c34d8d9a8bc530b5151b58"
I20260927 18:43:20.052910 20331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdd8431e72a40c4ac2e414f4873537e" candidate_uuid: "11a7e1d48ca947b68c1328540f0a8937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10115c8fe9f343f2b81ff466eee07d79"
I20260927 18:43:20.053181 20331 raft_consensus.cc:2379] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 11a7e1d48ca947b68c1328540f0a8937 in current term 1: Already voted for candidate 10115c8fe9f343f2b81ff466eee07d79 in this term.
I20260927 18:43:20.054260 20462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdd8431e72a40c4ac2e414f4873537e" candidate_uuid: "10115c8fe9f343f2b81ff466eee07d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11a7e1d48ca947b68c1328540f0a8937"
I20260927 18:43:20.054548 20462 raft_consensus.cc:2379] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10115c8fe9f343f2b81ff466eee07d79 in current term 1: Already voted for candidate 11a7e1d48ca947b68c1328540f0a8937 in this term.
I20260927 18:43:20.055125 20265 leader_election.cc:302] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10115c8fe9f343f2b81ff466eee07d79; no voters: 11a7e1d48ca947b68c1328540f0a8937, 4f55629a23c34d8d9a8bc530b5151b58
I20260927 18:43:20.055408 20665 raft_consensus.cc:2726] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 18:43:20.055685 20592 raft_consensus.cc:2454] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11a7e1d48ca947b68c1328540f0a8937 in term 1.
I20260927 18:43:20.056041 20398 leader_election.cc:302] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11a7e1d48ca947b68c1328540f0a8937, 4f55629a23c34d8d9a8bc530b5151b58; no voters: 10115c8fe9f343f2b81ff466eee07d79
I20260927 18:43:20.056378 20664 raft_consensus.cc:2781] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:20.056857 20664 raft_consensus.cc:686] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 1 LEADER]: Becoming Leader. State: Replica: 11a7e1d48ca947b68c1328540f0a8937, State: Running, Role: LEADER
I20260927 18:43:20.057425 20664 consensus_queue.cc:237] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 } } peers { permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 } } peers { permanent_uuid: "11a7e1d48ca947b68c1328540f0a8937" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39157 } }
I20260927 18:43:20.063182 20189 catalog_manager.cc:5495] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11a7e1d48ca947b68c1328540f0a8937 (127.2.95.66). New cstate: current_term: 1 leader_uuid: "11a7e1d48ca947b68c1328540f0a8937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11a7e1d48ca947b68c1328540f0a8937" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 39157 } health_report { overall_health: HEALTHY } } }
I20260927 18:43:20.094544 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:20.088432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53704 (local address 127.2.95.66:39157)
0927 18:43:20.088661 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:20.088668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:20.088689 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:43:20.088812 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:20.088817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:20.088912 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:20.089079 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:43:20.089495 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.090444 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.091934 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.092396 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.092525 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:20.093092 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:43:20.093837 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:43:20.093997 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:20.094102 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 18:43:20.094206 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:43:20.104874 20592 raft_consensus.cc:1261] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58 [term 1 FOLLOWER]: Refusing update from remote peer 11a7e1d48ca947b68c1328540f0a8937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:20.104889 20331 raft_consensus.cc:1261] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 1 FOLLOWER]: Refusing update from remote peer 11a7e1d48ca947b68c1328540f0a8937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:20.105846 20664 consensus_queue.cc:1035] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f55629a23c34d8d9a8bc530b5151b58" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:20.106194 20671 consensus_queue.cc:1035] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10115c8fe9f343f2b81ff466eee07d79" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 45445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:20.118307 20676 mvcc.cc:205] Tried to move back new op lower bound from 7334029722018148352 to 7334029721839665152. Current Snapshot: MvccSnapshot[applied={T|T < 7334029722018148352}]
I20260927 18:43:20.136795 20189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:20.137204 20189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:20.141888 20189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3090c1590bed4caaa000862d96d5e8e1: Sending DeleteTablet for 3 replicas of tablet 8bdd8431e72a40c4ac2e414f4873537e
I20260927 18:43:20.143347 20311 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bdd8431e72a40c4ac2e414f4873537e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:20 UTC) from {username='slave'} at 127.0.0.1:35910
I20260927 18:43:20.143352 20573 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bdd8431e72a40c4ac2e414f4873537e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:20 UTC) from {username='slave'} at 127.0.0.1:41226
I20260927 18:43:20.143465 20442 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bdd8431e72a40c4ac2e414f4873537e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:20 UTC) from {username='slave'} at 127.0.0.1:53698
I20260927 18:43:20.144449 20686 tablet_replica.cc:330] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79: stopping tablet replica
I20260927 18:43:20.144999 20686 raft_consensus.cc:2229] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:20.145617 20686 raft_consensus.cc:2258] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:20.148341 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 20249
I20260927 18:43:20.154342 20684 tablet_replica.cc:330] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937: stopping tablet replica
I20260927 18:43:20.155182 20684 raft_consensus.cc:2229] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:43:20.158278 20686 ts_tablet_manager.cc:1905] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:20.167910 20686 ts_tablet_manager.cc:1918] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:20.168310 20686 log.cc:1196] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/8bdd8431e72a40c4ac2e414f4873537e
I20260927 18:43:20.169070 20686 ts_tablet_manager.cc:1939] T 8bdd8431e72a40c4ac2e414f4873537e P 10115c8fe9f343f2b81ff466eee07d79: Deleting consensus metadata
I20260927 18:43:20.170258 20684 raft_consensus.cc:2258] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:20.171320 20685 tablet_replica.cc:330] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58: stopping tablet replica
I20260927 18:43:20.171860 20174 catalog_manager.cc:4841] TS 10115c8fe9f343f2b81ff466eee07d79 (127.2.95.65:45445): tablet 8bdd8431e72a40c4ac2e414f4873537e (table test-table [id=6e045b8cf3644da09a5fbe67f48518a4]) successfully deleted
I20260927 18:43:20.171969 20685 raft_consensus.cc:2229] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:20.172729 20685 raft_consensus.cc:2258] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:20.174563 20684 ts_tablet_manager.cc:1905] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:20.175110 20685 ts_tablet_manager.cc:1905] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:20.182873 20685 ts_tablet_manager.cc:1918] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:20.182978 20684 ts_tablet_manager.cc:1918] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:20.183151 20684 log.cc:1196] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/8bdd8431e72a40c4ac2e414f4873537e
I20260927 18:43:20.183290 20685 log.cc:1196] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/8bdd8431e72a40c4ac2e414f4873537e
I20260927 18:43:20.183786 20684 ts_tablet_manager.cc:1939] T 8bdd8431e72a40c4ac2e414f4873537e P 11a7e1d48ca947b68c1328540f0a8937: Deleting consensus metadata
I20260927 18:43:20.183969 20685 ts_tablet_manager.cc:1939] T 8bdd8431e72a40c4ac2e414f4873537e P 4f55629a23c34d8d9a8bc530b5151b58: Deleting consensus metadata
I20260927 18:43:20.186383 20177 catalog_manager.cc:4841] TS 4f55629a23c34d8d9a8bc530b5151b58 (127.2.95.67:38387): tablet 8bdd8431e72a40c4ac2e414f4873537e (table test-table [id=6e045b8cf3644da09a5fbe67f48518a4]) successfully deleted
I20260927 18:43:20.186875 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:20.150229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.2.95.65:45445)
0927 18:43:20.150448 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:20.150470 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:20.162505 (+ 12035us) server_negotiation.cc:407] Connection header received
0927 18:43:20.163100 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:20.163108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:20.163240 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:20.163317 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:20.164026 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.165320 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.167399 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.168129 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.168228 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:20.169934 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:20.169946 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:20.169968 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:20.170997 (+ 1029us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:20.171069 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:20.171102 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:20.171208 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:20.186238 (+ 15030us) server_negotiation.cc:299] Negotiation successful
0927 18:43:20.186454 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:43:20.187922 20175 catalog_manager.cc:4841] TS 11a7e1d48ca947b68c1328540f0a8937 (127.2.95.66:39157): tablet 8bdd8431e72a40c4ac2e414f4873537e (table test-table [id=6e045b8cf3644da09a5fbe67f48518a4]) successfully deleted
I20260927 18:43:20.188413 20371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:20.287818 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 20249
I20260927 18:43:20.314225 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 20380
I20260927 18:43:20.323639 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:20.315199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53712 (local address 127.2.95.66:39157)
0927 18:43:20.315421 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:20.315427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:20.315452 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:43:20.315621 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:20.315628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:20.315776 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:20.315899 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:20.316346 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.317636 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.319317 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.320010 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.320149 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:20.321436 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:20.321450 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:20.321474 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:20.322528 (+ 1054us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:20.322622 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:20.322651 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:20.322763 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:20.322949 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 18:43:20.323121 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 18:43:20.326819 20502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:20.456719 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 20380
I20260927 18:43:20.476599 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 20511
I20260927 18:43:20.484181 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:20.477530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.2.95.67:38387)
0927 18:43:20.477773 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:20.477779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:20.477800 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:43:20.477959 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:20.477963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:20.478087 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:20.478179 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:20.478637 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.479439 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.480900 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:20.481314 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:20.481407 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:20.482688 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:20.482699 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:20.482719 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:20.483419 (+ 700us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:20.483509 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:20.483532 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:20.483621 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:20.483750 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 18:43:20.483851 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 18:43:20.484953 20633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:20.582914 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 20511
I20260927 18:43:20.599750 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 20159
I20260927 18:43:20.601233 20220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:20.727351 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 20159
2026-09-27T18:43:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3357 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:20 dist-test-slave-9jll krb5kdc[20694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:20 dist-test-slave-9jll krb5kdc[20694](info): set up 2 sockets
Sep 27 18:43:20 dist-test-slave-9jll krb5kdc[20694](info): commencing operation
krb5kdc: starting...
W20260927 18:43:21.778398 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.004s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:21 dist-test-slave-9jll krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:43:22.077778 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:43335
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:43335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:22.473714 20707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:22.474107 20707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:22.474248 20707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:22.482259 20707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:22.482421 20707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:22.482477 20707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:22.482573 20707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:22.482631 20707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:22.493613 20707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:43335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:43335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:22.495419 20707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:22.497051 20707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:22.511018 20712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:22.510977 20713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:22.510946 20715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:22.512269 20707 server_base.cc:1034] running on GCE node
I20260927 18:43:22.513218 20707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:22.515116 20707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:22.516355 20707 hybrid_clock.cc:648] HybridClock initialized: now 1790534602516302 us; error 43 us; skew 500 ppm
I20260927 18:43:22.516736 20707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:22 dist-test-slave-9jll krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534602, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:22.528518 20707 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:43:22.530638 20707 webserver.cc:460] Webserver started at http://127.2.95.126:45323/ using document root <none> and password file <none>
I20260927 18:43:22.531517 20707 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:22.531641 20707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:22.531975 20707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:22.535732 20707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "234146aaa44a4df1a2623808912fe71f"
format_stamp: "Formatted at 2026-09-27 18:43:22 on dist-test-slave-9jll"
server_key: "f80cc8ff33560ff80495161ce9d84b64"
server_key_iv: "1742b4222f76c8031abbb8514619fbf2"
server_key_version: "encryptionkey@0"
I20260927 18:43:22.536569 20707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "234146aaa44a4df1a2623808912fe71f"
format_stamp: "Formatted at 2026-09-27 18:43:22 on dist-test-slave-9jll"
server_key: "f80cc8ff33560ff80495161ce9d84b64"
server_key_iv: "1742b4222f76c8031abbb8514619fbf2"
server_key_version: "encryptionkey@0"
I20260927 18:43:22.542979 20707 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 18:43:22.547161 20722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:22.548975 20707 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:43:22.549156 20707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "234146aaa44a4df1a2623808912fe71f"
format_stamp: "Formatted at 2026-09-27 18:43:22 on dist-test-slave-9jll"
server_key: "f80cc8ff33560ff80495161ce9d84b64"
server_key_iv: "1742b4222f76c8031abbb8514619fbf2"
server_key_version: "encryptionkey@0"
I20260927 18:43:22.549372 20707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:22.568706 20707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:22.573915 20707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:22.574213 20707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:22.595750 20707 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:43335
I20260927 18:43:22.595765 20773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:43335 every 8 connection(s)
I20260927 18:43:22.597595 20707 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:22.600183 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 20707
I20260927 18:43:22.600368 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:22.600915 2429 external_mini_cluster.cc:1428] Setting key d226e2d5197c25d22ebf3c36c3f2614e
I20260927 18:43:22.602972 20774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:43:22 dist-test-slave-9jll krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:43:22.618872 20774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f: Bootstrap starting.
I20260927 18:43:22.623775 20774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:22.625200 20774 log.cc:824] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:22.629245 20774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f: No bootstrap required, opened a new log
I20260927 18:43:22.629793 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:22.603339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.2.95.126:43335)
0927 18:43:22.604287 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:22.604305 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:22.604365 (+ 60us) server_negotiation.cc:407] Connection header received
0927 18:43:22.606796 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:22.606831 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:22.607500 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:22.608091 (+ 591us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:22.609855 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:22.612034 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:22.613791 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:22.614912 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:22.615128 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:22.619951 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:22.619963 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:22.619987 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:22.620043 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:22.624298 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:22.625207 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:22.625217 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:22.625227 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:22.625476 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:22.626116 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:22.626120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:22.626122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:22.626820 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:22.627148 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:22.627656 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 18:43:22.628299 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":410,"threads_started":1}
I20260927 18:43:22.638104 20774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "234146aaa44a4df1a2623808912fe71f" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43335 } }
I20260927 18:43:22.638646 20774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:22.638782 20774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 234146aaa44a4df1a2623808912fe71f, State: Initialized, Role: FOLLOWER
I20260927 18:43:22.639498 20774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "234146aaa44a4df1a2623808912fe71f" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43335 } }
I20260927 18:43:22.639751 20774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:22.639881 20774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:22.640067 20774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:22.642628 20774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "234146aaa44a4df1a2623808912fe71f" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43335 } }
I20260927 18:43:22.643409 20774 leader_election.cc:302] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 234146aaa44a4df1a2623808912fe71f; no voters:
I20260927 18:43:22.644246 20774 leader_election.cc:288] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:22.644363 20779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:22.645164 20779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [term 1 LEADER]: Becoming Leader. State: Replica: 234146aaa44a4df1a2623808912fe71f, State: Running, Role: LEADER
I20260927 18:43:22.645903 20779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "234146aaa44a4df1a2623808912fe71f" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43335 } }
I20260927 18:43:22.647135 20774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:22.649514 20781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 234146aaa44a4df1a2623808912fe71f. Latest consensus state: current_term: 1 leader_uuid: "234146aaa44a4df1a2623808912fe71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "234146aaa44a4df1a2623808912fe71f" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43335 } } }
I20260927 18:43:22.649573 20780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "234146aaa44a4df1a2623808912fe71f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "234146aaa44a4df1a2623808912fe71f" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43335 } } }
I20260927 18:43:22.650131 20780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:22.650902 20781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:22.653810 20785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:22.659324 20785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:22.669872 20785 catalog_manager.cc:1347] Generated new cluster ID: ff39674472774912bea60066566899a0
I20260927 18:43:22.670022 20785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:22.681468 20785 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:22.683845 20785 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:22.699554 20785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f: Generated new TSK 0
I20260927 18:43:22.700791 20785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:43:22.785871 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:43335
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:23.180495 20800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:23.180866 20800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:23.180981 20800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:23.189033 20800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:23.189190 20800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:23.189320 20800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:43:23.200815 20800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:43335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:23.202642 20800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:23.204315 20800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:23.219816 20808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:23.219887 20805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:23.219856 20806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:23.220813 20800 server_base.cc:1034] running on GCE node
I20260927 18:43:23.221714 20800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:23.223026 20800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:23.224251 20800 hybrid_clock.cc:648] HybridClock initialized: now 1790534603224192 us; error 49 us; skew 500 ppm
I20260927 18:43:23.224586 20800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:23 dist-test-slave-9jll krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:23.228089 20800 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:43:23.229897 20800 webserver.cc:460] Webserver started at http://127.2.95.65:35327/ using document root <none> and password file <none>
I20260927 18:43:23.230720 20800 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:23.230827 20800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:23.231141 20800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:23.234624 20800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "b85c09bbe2654dbd8cb3148678dd48cc"
format_stamp: "Formatted at 2026-09-27 18:43:23 on dist-test-slave-9jll"
server_key: "6b3dcc3b3a570ab93f305fb5f155aa79"
server_key_iv: "923098857084c89dbe11e86e2fa9cfbc"
server_key_version: "encryptionkey@0"
I20260927 18:43:23.235491 20800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "b85c09bbe2654dbd8cb3148678dd48cc"
format_stamp: "Formatted at 2026-09-27 18:43:23 on dist-test-slave-9jll"
server_key: "6b3dcc3b3a570ab93f305fb5f155aa79"
server_key_iv: "923098857084c89dbe11e86e2fa9cfbc"
server_key_version: "encryptionkey@0"
I20260927 18:43:23.241848 20800 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 18:43:23.246254 20815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:23.248059 20800 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:43:23.248255 20800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "b85c09bbe2654dbd8cb3148678dd48cc"
format_stamp: "Formatted at 2026-09-27 18:43:23 on dist-test-slave-9jll"
server_key: "6b3dcc3b3a570ab93f305fb5f155aa79"
server_key_iv: "923098857084c89dbe11e86e2fa9cfbc"
server_key_version: "encryptionkey@0"
I20260927 18:43:23.248489 20800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:23.271690 20800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:23.276844 20800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:23.277149 20800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:23.278369 20800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:23.279954 20800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:23.280035 20800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:23.280135 20800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:23.280181 20800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:23.316846 20800 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:43691
I20260927 18:43:23.316979 20927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:43691 every 8 connection(s)
I20260927 18:43:23.318456 20800 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:43:23.328084 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 20800
I20260927 18:43:23.328276 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:43:23.328776 2429 external_mini_cluster.cc:1428] Setting key 4117e611107d2093151a759fdb7f8053
Sep 27 18:43:23 dist-test-slave-9jll krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:43:23.341213 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:23.321589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:42911 (local address 127.2.95.126:43335)
0927 18:43:23.322119 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:23.322124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:23.323152 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 18:43:23.325249 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:23.325256 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:23.325372 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:23.325482 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:23.327264 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:23.328095 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:23.330034 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:23.330496 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:23.330617 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:23.334682 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:23.334688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:23.334691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:23.334730 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:23.337751 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:23.338608 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:23.338613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:23.338616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:23.338763 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:23.339211 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:23.339216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:23.339218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:23.339527 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:23.339671 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:23.340527 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 18:43:23.340809 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":283,"threads_started":1}
I20260927 18:43:23.342489 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:23.322133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:43335 (local address 127.2.95.65:42911)
0927 18:43:23.322964 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:23.323016 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:23.324943 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:23.325799 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:23.325808 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:23.326410 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:23.327070 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:23.327089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:23.328307 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:23.328322 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:23.329887 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:23.329901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:23.330735 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:23.330744 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:23.330988 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:23.331760 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:23.331786 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:23.334416 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:23.337901 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:23.337912 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:23.337934 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:23.338469 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:23.338879 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:23.338884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:23.338888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:23.339091 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:23.339657 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:23.339666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:23.340079 (+ 413us) client_negotiation.cc:769] Sending connection context
0927 18:43:23.340371 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 18:43:23.341039 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":298,"threads_started":1}
I20260927 18:43:23.344439 20928 heartbeater.cc:347] Connected to a master server at 127.2.95.126:43335
I20260927 18:43:23.344796 20928 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:23.345762 20928 heartbeater.cc:511] Master 127.2.95.126:43335 requested a full tablet report, sending...
I20260927 18:43:23.348765 20739 ts_manager.cc:194] Registered new tserver with Master: b85c09bbe2654dbd8cb3148678dd48cc (127.2.95.65:43691)
I20260927 18:43:23.351065 20739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:42911
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:43:23.399080 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:43335
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:23.787917 20934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:23.788300 20934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:23.788453 20934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:23.796325 20934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:23.796494 20934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:23.796656 20934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:43:23.808027 20934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:43335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:23.810053 20934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:23.811789 20934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:23.827227 20939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:23.827256 20942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:23.827298 20940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:23.828186 20934 server_base.cc:1034] running on GCE node
I20260927 18:43:23.828969 20934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:23.830394 20934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:23.831661 20934 hybrid_clock.cc:648] HybridClock initialized: now 1790534603831600 us; error 57 us; skew 500 ppm
I20260927 18:43:23.832054 20934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:23 dist-test-slave-9jll krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:23.835750 20934 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:43:23.837831 20934 webserver.cc:460] Webserver started at http://127.2.95.66:46545/ using document root <none> and password file <none>
I20260927 18:43:23.838703 20934 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:23.838829 20934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:23.839174 20934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:23.842867 20934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "52fe1925c1c84d83b13ff35032a0b630"
format_stamp: "Formatted at 2026-09-27 18:43:23 on dist-test-slave-9jll"
server_key: "626d756eef511bb3619e6efaf6499270"
server_key_iv: "c94c359dc2be8d907575e26568eae71e"
server_key_version: "encryptionkey@0"
I20260927 18:43:23.843837 20934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "52fe1925c1c84d83b13ff35032a0b630"
format_stamp: "Formatted at 2026-09-27 18:43:23 on dist-test-slave-9jll"
server_key: "626d756eef511bb3619e6efaf6499270"
server_key_iv: "c94c359dc2be8d907575e26568eae71e"
server_key_version: "encryptionkey@0"
I20260927 18:43:23.850332 20934 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 18:43:23.854884 20949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:23.856606 20934 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:43:23.856781 20934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "52fe1925c1c84d83b13ff35032a0b630"
format_stamp: "Formatted at 2026-09-27 18:43:23 on dist-test-slave-9jll"
server_key: "626d756eef511bb3619e6efaf6499270"
server_key_iv: "c94c359dc2be8d907575e26568eae71e"
server_key_version: "encryptionkey@0"
I20260927 18:43:23.857004 20934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:23.886937 20934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:23.893361 20934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:23.893695 20934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:23.894883 20934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:23.896476 20934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:23.896552 20934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:23.896651 20934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:23.896695 20934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:23.940779 20934 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:38211
I20260927 18:43:23.940876 21061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:38211 every 8 connection(s)
I20260927 18:43:23.943140 20934 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:43:23.951591 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 20934
I20260927 18:43:23.951833 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:43:23.952526 2429 external_mini_cluster.cc:1428] Setting key 48475f44c57b31994bb444d0dc63b85a
Sep 27 18:43:23 dist-test-slave-9jll krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:43:23.975927 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:23.948272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:50699 (local address 127.2.95.126:43335)
0927 18:43:23.948807 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:23.948815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:23.949070 (+ 255us) server_negotiation.cc:407] Connection header received
0927 18:43:23.952349 (+ 3279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:23.952358 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:23.952484 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:23.952603 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:23.955801 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:23.956946 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:23.959698 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:23.960459 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:23.960613 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:23.966324 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:23.966333 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:23.966339 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:23.966398 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:23.970731 (+ 4333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:23.971937 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:23.971944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:23.971948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:23.972131 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:23.972741 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:23.972746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:23.972749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:23.973187 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:23.973359 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:23.975176 (+ 1817us) server_negotiation.cc:299] Negotiation successful
0927 18:43:23.975490 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":1445,"threads_started":1}
I20260927 18:43:23.977018 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:23.947637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:43335 (local address 127.2.95.66:50699)
0927 18:43:23.948830 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:23.948901 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:23.951815 (+ 2914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:23.953502 (+ 1687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:23.953518 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:23.954456 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:23.955528 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:23.955563 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:23.957155 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:23.957162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:23.959481 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:23.959502 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:23.960775 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:23.960783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:23.961209 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:23.962210 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:23.962251 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:23.965887 (+ 3636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:23.970954 (+ 5067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:23.970966 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:23.970993 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:23.971752 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:23.972266 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:23.972271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:23.972275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:23.972603 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:23.973332 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:23.973343 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:23.973939 (+ 596us) client_negotiation.cc:769] Sending connection context
0927 18:43:23.974307 (+ 368us) client_negotiation.cc:240] Negotiation successful
0927 18:43:23.974994 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":679,"threads_started":1}
I20260927 18:43:23.978963 21062 heartbeater.cc:347] Connected to a master server at 127.2.95.126:43335
I20260927 18:43:23.979351 21062 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:23.980333 21062 heartbeater.cc:511] Master 127.2.95.126:43335 requested a full tablet report, sending...
I20260927 18:43:23.982266 20739 ts_manager.cc:194] Registered new tserver with Master: 52fe1925c1c84d83b13ff35032a0b630 (127.2.95.66:38211)
I20260927 18:43:23.983348 20739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:50699
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:43:24.026885 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:43335
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:43:24.355536 20928 heartbeater.cc:503] Master 127.2.95.126:43335 was elected leader, sending a full tablet report...
W20260927 18:43:24.419147 21068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:24.419562 21068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:24.419704 21068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:24.428021 21068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:24.428186 21068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:24.428354 21068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:43:24.439798 21068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:43335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:24.441576 21068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:24.443316 21068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:24.459558 21076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:24.459633 21074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:24.459637 21073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:24.460673 21068 server_base.cc:1034] running on GCE node
I20260927 18:43:24.461465 21068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:24.462694 21068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:24.463961 21068 hybrid_clock.cc:648] HybridClock initialized: now 1790534604463903 us; error 45 us; skew 500 ppm
I20260927 18:43:24.464350 21068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:24 dist-test-slave-9jll krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:24.468048 21068 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:43:24.469916 21068 webserver.cc:460] Webserver started at http://127.2.95.67:45475/ using document root <none> and password file <none>
I20260927 18:43:24.470789 21068 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:24.470913 21068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:24.471228 21068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:24.474947 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "7bc6d194ffa44bafa6465caac44c35a5"
format_stamp: "Formatted at 2026-09-27 18:43:24 on dist-test-slave-9jll"
server_key: "9ad9c071b06f8d30ae69a840b21944d9"
server_key_iv: "7aee270cf54e8e2df1f5c187cd771787"
server_key_version: "encryptionkey@0"
I20260927 18:43:24.475847 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "7bc6d194ffa44bafa6465caac44c35a5"
format_stamp: "Formatted at 2026-09-27 18:43:24 on dist-test-slave-9jll"
server_key: "9ad9c071b06f8d30ae69a840b21944d9"
server_key_iv: "7aee270cf54e8e2df1f5c187cd771787"
server_key_version: "encryptionkey@0"
I20260927 18:43:24.482372 21068 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 18:43:24.486950 21083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:24.488756 21068 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:43:24.488982 21068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "7bc6d194ffa44bafa6465caac44c35a5"
format_stamp: "Formatted at 2026-09-27 18:43:24 on dist-test-slave-9jll"
server_key: "9ad9c071b06f8d30ae69a840b21944d9"
server_key_iv: "7aee270cf54e8e2df1f5c187cd771787"
server_key_version: "encryptionkey@0"
I20260927 18:43:24.489238 21068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:24.507328 21068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:24.512643 21068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:24.512982 21068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:24.514199 21068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:24.515856 21068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:24.515946 21068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:24.516083 21068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:24.516134 21068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:24.551725 21068 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:44033
I20260927 18:43:24.551749 21195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:44033 every 8 connection(s)
I20260927 18:43:24.553375 21068 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:43:24.559126 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 21068
I20260927 18:43:24.559317 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:43:24.559794 2429 external_mini_cluster.cc:1428] Setting key b0f3ea5b9a45a71a8443826a98336ef3
Sep 27 18:43:24 dist-test-slave-9jll krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:43:24.579027 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.556599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:33175 (local address 127.2.95.126:43335)
0927 18:43:24.557201 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:24.557206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:24.558200 (+ 994us) server_negotiation.cc:407] Connection header received
0927 18:43:24.560969 (+ 2769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:24.560974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:24.561053 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:24.561153 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:24.563725 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.564614 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.566407 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.566849 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.566959 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.571409 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:24.571416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:24.571420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:24.571465 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:24.574777 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:24.575843 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:24.575849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:24.575853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:24.576034 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:24.576686 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:24.576690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:24.576692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:24.577065 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:24.577199 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:24.578378 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0927 18:43:24.578601 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":308,"threads_started":1}
I20260927 18:43:24.580334 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.557162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:43335 (local address 127.2.95.67:33175)
0927 18:43:24.558014 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:24.558059 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:24.560145 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:24.561552 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:24.561564 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:24.562481 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:24.563531 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:24.563559 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.564748 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.564752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:24.566232 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:24.566247 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.567063 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.567068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:24.567358 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.568196 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:24.568225 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:24.571065 (+ 2840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:24.574963 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:24.574977 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:24.575004 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:24.575717 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:24.576187 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:24.576192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:24.576196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:24.576540 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:24.577256 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:24.577268 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:24.577882 (+ 614us) client_negotiation.cc:769] Sending connection context
0927 18:43:24.578196 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 18:43:24.578757 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":306,"threads_started":1}
I20260927 18:43:24.582185 21196 heartbeater.cc:347] Connected to a master server at 127.2.95.126:43335
I20260927 18:43:24.582588 21196 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:24.583528 21196 heartbeater.cc:511] Master 127.2.95.126:43335 requested a full tablet report, sending...
I20260927 18:43:24.585481 20739 ts_manager.cc:194] Registered new tserver with Master: 7bc6d194ffa44bafa6465caac44c35a5 (127.2.95.67:44033)
I20260927 18:43:24.586673 20739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:33175
I20260927 18:43:24.589943 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:43:24.613638 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.595466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.2.95.126:43335)
0927 18:43:24.595729 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:24.595737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:24.596154 (+ 417us) server_negotiation.cc:407] Connection header received
0927 18:43:24.596359 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:24.596365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:24.596450 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:24.596550 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:24.598614 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.599554 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.601967 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.602480 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.602612 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.606584 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:24.606589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:24.606593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:24.606642 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:24.609642 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:24.610962 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:24.610967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:24.610970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:24.611135 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:24.611700 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:24.611705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:24.611707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:24.612183 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:24.612438 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:24.613050 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 18:43:24.613272 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:43:24.618844 20739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:43:24.626782 20739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:43:24.656570 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.643990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38476 (local address 127.2.95.66:38211)
0927 18:43:24.646621 (+ 2631us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:24.646632 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:24.646666 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:43:24.646786 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:24.646796 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:24.647055 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:24.648433 (+ 1378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:24.648684 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.650065 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.653462 (+ 3397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.655180 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.655322 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.655560 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:24.656004 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 18:43:24.656203 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2416,"thread_start_us":2235,"threads_started":1}
I20260927 18:43:24.656694 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.643317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:38211 (local address 127.0.0.1:38476)
0927 18:43:24.644519 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:24.644536 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:24.644710 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:24.647336 (+ 2626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:24.647346 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:24.647399 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:24.648079 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:24.648097 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.650195 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.650200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:24.653128 (+ 2928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:24.653145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.655396 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.655403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:24.655659 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.655737 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 18:43:24.655921 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 18:43:24.656155 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":114,"thread_start_us":1227,"threads_started":1}
I20260927 18:43:24.657481 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.642502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:44033 (local address 127.0.0.1:50050)
0927 18:43:24.643646 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:24.643697 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:24.643941 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:24.649271 (+ 5330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:24.649277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:24.649305 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:24.649708 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:24.649721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.651676 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.651681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:24.653865 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:24.653877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.655476 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.655482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:24.655949 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.655981 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 18:43:24.656037 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 18:43:24.656153 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":2,"thread_start_us":612,"threads_started":1}
I20260927 18:43:24.657487 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.642881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50050 (local address 127.2.95.67:44033)
0927 18:43:24.648552 (+ 5671us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:24.648562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:24.648602 (+ 40us) server_negotiation.cc:407] Connection header received
0927 18:43:24.648721 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:24.648730 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:24.649024 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:24.649248 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:24.649891 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.651487 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.654024 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.655256 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.656293 (+ 1037us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.656531 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:24.656610 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 18:43:24.656908 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5469,"thread_start_us":5331,"threads_started":1}
I20260927 18:43:24.658234 21212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.643668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:43691 (local address 127.0.0.1:37546)
0927 18:43:24.646087 (+ 2419us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:24.646106 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:24.646360 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:24.647565 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:24.647569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:24.647628 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:24.648256 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:24.648271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.650230 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.650235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:24.653250 (+ 3015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:24.653262 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.657280 (+ 4018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.657287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:24.657485 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.657518 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 18:43:24.657615 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 18:43:24.657815 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2282,"mutex_wait_us":376,"thread_start_us":2128,"threads_started":1}
I20260927 18:43:24.659798 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.644446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.2.95.65:43691)
0927 18:43:24.645499 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:24.645512 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:24.646224 (+ 712us) server_negotiation.cc:407] Connection header received
0927 18:43:24.646611 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:24.646622 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:24.646942 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:24.647225 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:24.648486 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.650084 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.653538 (+ 3454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.655180 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.655322 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.655596 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:24.658085 (+ 2489us) server_negotiation.cc:299] Negotiation successful
0927 18:43:24.658485 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":515,"threads_started":1}
I20260927 18:43:24.662353 20997 tablet_service.cc:1467] Processing CreateTablet for tablet b000c19667b64d97b22c3a92c67b14a5 (DEFAULT_TABLE table=test-table [id=e2dfdb6e90ea42eda1e4fb2d03545a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:24.662353 21131 tablet_service.cc:1467] Processing CreateTablet for tablet b000c19667b64d97b22c3a92c67b14a5 (DEFAULT_TABLE table=test-table [id=e2dfdb6e90ea42eda1e4fb2d03545a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:24.664131 21131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b000c19667b64d97b22c3a92c67b14a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:24.664142 20997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b000c19667b64d97b22c3a92c67b14a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:24.663775 20863 tablet_service.cc:1467] Processing CreateTablet for tablet b000c19667b64d97b22c3a92c67b14a5 (DEFAULT_TABLE table=test-table [id=e2dfdb6e90ea42eda1e4fb2d03545a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:24.664987 20863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b000c19667b64d97b22c3a92c67b14a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:24.677999 21215 tablet_bootstrap.cc:492] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc: Bootstrap starting.
I20260927 18:43:24.680751 21216 tablet_bootstrap.cc:492] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5: Bootstrap starting.
I20260927 18:43:24.682716 21217 tablet_bootstrap.cc:492] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630: Bootstrap starting.
I20260927 18:43:24.684818 21216 tablet_bootstrap.cc:654] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:24.684818 21215 tablet_bootstrap.cc:654] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:24.686093 21217 tablet_bootstrap.cc:654] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:24.686399 21216 log.cc:824] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:24.686385 21215 log.cc:824] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:24.687279 21217 log.cc:824] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:24.690320 21216 tablet_bootstrap.cc:492] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5: No bootstrap required, opened a new log
I20260927 18:43:24.690514 21217 tablet_bootstrap.cc:492] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630: No bootstrap required, opened a new log
I20260927 18:43:24.690678 21216 ts_tablet_manager.cc:1397] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260927 18:43:24.690810 21217 ts_tablet_manager.cc:1397] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260927 18:43:24.691601 21215 tablet_bootstrap.cc:492] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc: No bootstrap required, opened a new log
I20260927 18:43:24.691972 21215 ts_tablet_manager.cc:1397] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260927 18:43:24.699961 21216 raft_consensus.cc:348] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc6d194ffa44bafa6465caac44c35a5" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44033 } } peers { permanent_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 43691 } } peers { permanent_uuid: "52fe1925c1c84d83b13ff35032a0b630" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38211 } }
I20260927 18:43:24.700482 21216 raft_consensus.cc:374] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:24.700593 21216 raft_consensus.cc:729] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc6d194ffa44bafa6465caac44c35a5, State: Initialized, Role: FOLLOWER
I20260927 18:43:24.701465 21216 consensus_queue.cc:260] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc6d194ffa44bafa6465caac44c35a5" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44033 } } peers { permanent_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 43691 } } peers { permanent_uuid: "52fe1925c1c84d83b13ff35032a0b630" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38211 } }
I20260927 18:43:24.702665 21217 raft_consensus.cc:348] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc6d194ffa44bafa6465caac44c35a5" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44033 } } peers { permanent_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 43691 } } peers { permanent_uuid: "52fe1925c1c84d83b13ff35032a0b630" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38211 } }
I20260927 18:43:24.703212 21217 raft_consensus.cc:374] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:24.703327 21217 raft_consensus.cc:729] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fe1925c1c84d83b13ff35032a0b630, State: Initialized, Role: FOLLOWER
I20260927 18:43:24.703584 21215 raft_consensus.cc:348] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc6d194ffa44bafa6465caac44c35a5" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44033 } } peers { permanent_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 43691 } } peers { permanent_uuid: "52fe1925c1c84d83b13ff35032a0b630" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38211 } }
I20260927 18:43:24.704057 21215 raft_consensus.cc:374] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:24.704164 21215 raft_consensus.cc:729] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85c09bbe2654dbd8cb3148678dd48cc, State: Initialized, Role: FOLLOWER
I20260927 18:43:24.704712 21196 heartbeater.cc:503] Master 127.2.95.126:43335 was elected leader, sending a full tablet report...
I20260927 18:43:24.704890 21215 consensus_queue.cc:260] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc6d194ffa44bafa6465caac44c35a5" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44033 } } peers { permanent_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 43691 } } peers { permanent_uuid: "52fe1925c1c84d83b13ff35032a0b630" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38211 } }
I20260927 18:43:24.705315 21216 ts_tablet_manager.cc:1428] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260927 18:43:24.704195 21217 consensus_queue.cc:260] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc6d194ffa44bafa6465caac44c35a5" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44033 } } peers { permanent_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 43691 } } peers { permanent_uuid: "52fe1925c1c84d83b13ff35032a0b630" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38211 } }
I20260927 18:43:24.706622 21062 heartbeater.cc:503] Master 127.2.95.126:43335 was elected leader, sending a full tablet report...
I20260927 18:43:24.707319 21217 ts_tablet_manager.cc:1428] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260927 18:43:24.708662 21215 ts_tablet_manager.cc:1428] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc: Time spent starting tablet: real 0.017s user 0.013s sys 0.002s
W20260927 18:43:24.805601 21197 tablet.cc:2369] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:43:24.821559 20929 tablet.cc:2369] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:43:24.856676 21222 raft_consensus.cc:482] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:43:24.857151 21222 raft_consensus.cc:504] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc6d194ffa44bafa6465caac44c35a5" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44033 } } peers { permanent_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 43691 } } peers { permanent_uuid: "52fe1925c1c84d83b13ff35032a0b630" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38211 } }
I20260927 18:43:24.859531 21222 leader_election.cc:288] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bc6d194ffa44bafa6465caac44c35a5 (127.2.95.67:44033), 52fe1925c1c84d83b13ff35032a0b630 (127.2.95.66:38211)
I20260927 18:43:24.867185 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.860017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:54977 (local address 127.2.95.67:44033)
0927 18:43:24.860365 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:24.860373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:24.861310 (+ 937us) server_negotiation.cc:407] Connection header received
0927 18:43:24.861499 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:24.861504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:24.861580 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:24.861660 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:24.862376 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.863216 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.865204 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.866115 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.866237 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.866362 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:24.866726 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 18:43:24.866881 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 18:43:24.867184 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.860286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:45839 (local address 127.2.95.66:38211)
0927 18:43:24.860593 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:24.860613 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:24.861102 (+ 489us) server_negotiation.cc:407] Connection header received
0927 18:43:24.861370 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:24.861378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:24.861461 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:24.861553 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:24.862265 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.863123 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.865180 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.866093 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.866224 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.866350 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:24.866720 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 18:43:24.866872 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260927 18:43:24.867401 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.860019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:38211 (local address 127.2.95.65:45839)
0927 18:43:24.860971 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:24.860991 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:24.861179 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:24.861583 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:24.861588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:24.861615 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:24.862096 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:24.862108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.863271 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.863275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:24.865003 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:24.865018 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.866348 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.866357 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:24.866550 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.866580 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 18:43:24.866669 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 18:43:24.866883 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":65,"thread_start_us":808,"threads_started":1}
I20260927 18:43:24.867426 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.860045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:44033 (local address 127.2.95.65:54977)
0927 18:43:24.861184 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:24.861199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:24.861351 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:24.861733 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:24.861738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:24.861781 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:24.862187 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:24.862196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.863367 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.863371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:24.865001 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:24.865043 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.866397 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.866404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:24.866562 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.866587 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 18:43:24.866669 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 18:43:24.866888 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":74,"thread_start_us":494,"threads_started":1}
I20260927 18:43:24.868793 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b000c19667b64d97b22c3a92c67b14a5" candidate_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc6d194ffa44bafa6465caac44c35a5" is_pre_election: true
I20260927 18:43:24.868789 21017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b000c19667b64d97b22c3a92c67b14a5" candidate_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fe1925c1c84d83b13ff35032a0b630" is_pre_election: true
I20260927 18:43:24.869393 21151 raft_consensus.cc:2454] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b85c09bbe2654dbd8cb3148678dd48cc in term 0.
I20260927 18:43:24.869391 21017 raft_consensus.cc:2454] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b85c09bbe2654dbd8cb3148678dd48cc in term 0.
I20260927 18:43:24.870196 20816 leader_election.cc:302] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bc6d194ffa44bafa6465caac44c35a5, b85c09bbe2654dbd8cb3148678dd48cc; no voters:
I20260927 18:43:24.870675 21222 raft_consensus.cc:2781] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:43:24.870869 21222 raft_consensus.cc:482] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:43:24.870985 21222 raft_consensus.cc:3037] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:24.873541 21222 raft_consensus.cc:504] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc6d194ffa44bafa6465caac44c35a5" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44033 } } peers { permanent_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 43691 } } peers { permanent_uuid: "52fe1925c1c84d83b13ff35032a0b630" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38211 } }
I20260927 18:43:24.874315 21222 leader_election.cc:288] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [CANDIDATE]: Term 1 election: Requested vote from peers 7bc6d194ffa44bafa6465caac44c35a5 (127.2.95.67:44033), 52fe1925c1c84d83b13ff35032a0b630 (127.2.95.66:38211)
I20260927 18:43:24.874904 21017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b000c19667b64d97b22c3a92c67b14a5" candidate_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fe1925c1c84d83b13ff35032a0b630"
I20260927 18:43:24.874961 21151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b000c19667b64d97b22c3a92c67b14a5" candidate_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc6d194ffa44bafa6465caac44c35a5"
I20260927 18:43:24.875128 21017 raft_consensus.cc:3037] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:24.875200 21151 raft_consensus.cc:3037] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:24.877647 21017 raft_consensus.cc:2454] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85c09bbe2654dbd8cb3148678dd48cc in term 1.
I20260927 18:43:24.877684 21151 raft_consensus.cc:2454] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85c09bbe2654dbd8cb3148678dd48cc in term 1.
I20260927 18:43:24.878122 20816 leader_election.cc:302] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bc6d194ffa44bafa6465caac44c35a5, b85c09bbe2654dbd8cb3148678dd48cc; no voters:
I20260927 18:43:24.878450 21222 raft_consensus.cc:2781] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:24.878988 21222 raft_consensus.cc:686] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 1 LEADER]: Becoming Leader. State: Replica: b85c09bbe2654dbd8cb3148678dd48cc, State: Running, Role: LEADER
I20260927 18:43:24.879562 21222 consensus_queue.cc:237] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc6d194ffa44bafa6465caac44c35a5" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44033 } } peers { permanent_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 43691 } } peers { permanent_uuid: "52fe1925c1c84d83b13ff35032a0b630" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38211 } }
I20260927 18:43:24.885416 20737 catalog_manager.cc:5495] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc reported cstate change: term changed from 0 to 1, leader changed from <none> to b85c09bbe2654dbd8cb3148678dd48cc (127.2.95.65). New cstate: current_term: 1 leader_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc6d194ffa44bafa6465caac44c35a5" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85c09bbe2654dbd8cb3148678dd48cc" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 43691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52fe1925c1c84d83b13ff35032a0b630" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38211 } health_report { overall_health: UNKNOWN } } }
I20260927 18:43:24.908829 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.903059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.2.95.65:43691)
0927 18:43:24.903271 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:24.903276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:24.903296 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:43:24.903436 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:24.903441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:24.903534 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:24.903672 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:43:24.904091 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.904840 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.906412 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:24.906854 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:24.906966 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:24.907499 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:43:24.908185 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:43:24.908335 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:24.908450 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 18:43:24.908505 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 18:43:24.920137 21151 raft_consensus.cc:1261] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5 [term 1 FOLLOWER]: Refusing update from remote peer b85c09bbe2654dbd8cb3148678dd48cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:24.920137 21017 raft_consensus.cc:1261] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630 [term 1 FOLLOWER]: Refusing update from remote peer b85c09bbe2654dbd8cb3148678dd48cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:24.921308 21226 consensus_queue.cc:1035] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc6d194ffa44bafa6465caac44c35a5" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:24.921752 21222 consensus_queue.cc:1035] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "52fe1925c1c84d83b13ff35032a0b630" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 38211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:24.933096 21229 mvcc.cc:205] Tried to move back new op lower bound from 7334029741738430464 to 7334029741591244800. Current Snapshot: MvccSnapshot[applied={T|T < 7334029741738430464}]
I20260927 18:43:24.953303 21231 mvcc.cc:205] Tried to move back new op lower bound from 7334029741738430464 to 7334029741591244800. Current Snapshot: MvccSnapshot[applied={T|T < 7334029741738430464}]
I20260927 18:43:24.965073 20737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:24.965471 20737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:24.970446 20737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 234146aaa44a4df1a2623808912fe71f: Sending DeleteTablet for 3 replicas of tablet b000c19667b64d97b22c3a92c67b14a5
I20260927 18:43:24.971866 21131 tablet_service.cc:1514] Processing DeleteTablet for tablet b000c19667b64d97b22c3a92c67b14a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:24 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:50050
I20260927 18:43:24.971933 20863 tablet_service.cc:1514] Processing DeleteTablet for tablet b000c19667b64d97b22c3a92c67b14a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:24 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:37546
I20260927 18:43:24.973307 20997 tablet_service.cc:1514] Processing DeleteTablet for tablet b000c19667b64d97b22c3a92c67b14a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:24 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:38476
I20260927 18:43:24.976655 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 20800
I20260927 18:43:24.984782 21241 tablet_replica.cc:330] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630: stopping tablet replica
I20260927 18:43:24.985702 21241 raft_consensus.cc:2229] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:24.986515 21241 raft_consensus.cc:2258] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:24.987769 21239 tablet_replica.cc:330] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc: stopping tablet replica
I20260927 18:43:24.988365 21240 tablet_replica.cc:330] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5: stopping tablet replica
I20260927 18:43:24.988811 21240 raft_consensus.cc:2229] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:24.989353 21240 raft_consensus.cc:2258] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:24.990067 21239 raft_consensus.cc:2229] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:43:24.990761 21239 raft_consensus.cc:2258] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:24.993422 21240 ts_tablet_manager.cc:1905] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:24.993465 21239 ts_tablet_manager.cc:1905] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:25.000288 21241 ts_tablet_manager.cc:1905] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:25.001757 21240 ts_tablet_manager.cc:1918] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:25.001794 21239 ts_tablet_manager.cc:1918] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:25.002013 21239 log.cc:1196] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/b000c19667b64d97b22c3a92c67b14a5
I20260927 18:43:25.002084 21240 log.cc:1196] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/b000c19667b64d97b22c3a92c67b14a5
I20260927 18:43:25.002705 21239 ts_tablet_manager.cc:1939] T b000c19667b64d97b22c3a92c67b14a5 P b85c09bbe2654dbd8cb3148678dd48cc: Deleting consensus metadata
I20260927 18:43:25.002744 21240 ts_tablet_manager.cc:1939] T b000c19667b64d97b22c3a92c67b14a5 P 7bc6d194ffa44bafa6465caac44c35a5: Deleting consensus metadata
I20260927 18:43:25.005023 20726 catalog_manager.cc:4841] TS b85c09bbe2654dbd8cb3148678dd48cc (127.2.95.65:43691): tablet b000c19667b64d97b22c3a92c67b14a5 (table test-table [id=e2dfdb6e90ea42eda1e4fb2d03545a5f]) successfully deleted
I20260927 18:43:25.004971 20723 catalog_manager.cc:4841] TS 7bc6d194ffa44bafa6465caac44c35a5 (127.2.95.67:44033): tablet b000c19667b64d97b22c3a92c67b14a5 (table test-table [id=e2dfdb6e90ea42eda1e4fb2d03545a5f]) successfully deleted
Sep 27 18:43:25 dist-test-slave-9jll krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:43:25.013845 21241 ts_tablet_manager.cc:1918] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:25.014158 21241 log.cc:1196] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/b000c19667b64d97b22c3a92c67b14a5
I20260927 18:43:25.014864 21241 ts_tablet_manager.cc:1939] T b000c19667b64d97b22c3a92c67b14a5 P 52fe1925c1c84d83b13ff35032a0b630: Deleting consensus metadata
I20260927 18:43:25.017421 20725 catalog_manager.cc:4841] TS 52fe1925c1c84d83b13ff35032a0b630 (127.2.95.66:38211): tablet b000c19667b64d97b22c3a92c67b14a5 (table test-table [id=e2dfdb6e90ea42eda1e4fb2d03545a5f]) successfully deleted
I20260927 18:43:25.018819 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:24.978094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.2.95.65:43691)
0927 18:43:24.978343 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:24.978350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:24.994706 (+ 16356us) server_negotiation.cc:407] Connection header received
0927 18:43:24.995354 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:24.995361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:24.995475 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:24.995572 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:25.000709 (+ 5137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:25.002375 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:25.005159 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:25.005874 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:25.006008 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:25.011194 (+ 5186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:25.011256 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:25.011280 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:25.011330 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:25.014807 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:25.015748 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:25.015758 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:25.015761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:25.015927 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:25.016372 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:25.016377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:25.016379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:25.016837 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:25.017181 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:25.018213 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0927 18:43:25.018462 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:43:25.019598 20923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:25.133275 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 20800
I20260927 18:43:25.152661 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 20934
Sep 27 18:43:25 dist-test-slave-9jll krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:43:25.170967 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:25.153587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.2.95.66:38211)
0927 18:43:25.153804 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:25.153811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:25.153891 (+ 80us) server_negotiation.cc:407] Connection header received
0927 18:43:25.154081 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:25.154088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:25.154198 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:25.154310 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:25.156068 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:25.156982 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:25.158506 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:25.158977 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:25.159093 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:25.164440 (+ 5347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:25.164462 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:25.164485 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:25.164544 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:25.167919 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:25.168781 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:25.168791 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:25.168793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:25.168959 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:25.169295 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:25.169299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:25.169302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:25.169874 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:25.170198 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:25.170450 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 18:43:25.170660 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 18:43:25.171813 21057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:25.269797 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 20934
I20260927 18:43:25.286482 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 21068
Sep 27 18:43:25 dist-test-slave-9jll krb5kdc[20694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:43:25.304266 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:25.287457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.2.95.67:44033)
0927 18:43:25.287709 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:25.287715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:25.287738 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:43:25.287854 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:25.287858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:25.287937 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:25.288028 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:25.289623 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:25.290556 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:25.291918 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:25.292337 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:25.292441 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:25.297645 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:25.297702 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:25.297789 (+ 87us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:25.297845 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:25.300897 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:25.301814 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:25.301825 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:25.301828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:25.302014 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:25.302459 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:25.302463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:25.302465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:25.302940 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:25.303195 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:25.303585 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 18:43:25.303877 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 18:43:25.305083 21191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:25.404140 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 21068
I20260927 18:43:25.421104 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 20707
I20260927 18:43:25.422573 20769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:25.554616 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 20707
2026-09-27T18:43:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4828 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T18:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:25Z Disabled control of system clock
I20260927 18:43:25.642510 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:34129
--webserver_interface=127.2.95.126
--webserver_port=0
--builtin_ntp_servers=127.2.95.84:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:34129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:43:26.052922 21250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:26.053381 21250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:26.053490 21250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:26.061597 21250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:26.061775 21250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:26.061846 21250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:26.061940 21250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:26.062012 21250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:26.072377 21250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:34129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:34129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:26.074280 21250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:26.076476 21250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:26.090376 21255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:26.090353 21256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:26.090304 21258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:26.091311 21250 server_base.cc:1034] running on GCE node
I20260927 18:43:26.092300 21250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:26.094225 21250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:26.095482 21250 hybrid_clock.cc:648] HybridClock initialized: now 1790534606095415 us; error 44 us; skew 500 ppm
I20260927 18:43:26.095924 21250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:43:26.098803 21250 webserver.cc:460] Webserver started at http://127.2.95.126:40759/ using document root <none> and password file <none>
I20260927 18:43:26.099728 21250 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:26.099835 21250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:26.100212 21250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:26.103981 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "dd6943365fc548a29df3f4b53a0b8f74"
format_stamp: "Formatted at 2026-09-27 18:43:26 on dist-test-slave-9jll"
server_key: "297a462578da96a7cc98ef864c367f02"
server_key_iv: "3b648b62426e3d7fe147f834f93f2c71"
server_key_version: "encryptionkey@0"
I20260927 18:43:26.104976 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "dd6943365fc548a29df3f4b53a0b8f74"
format_stamp: "Formatted at 2026-09-27 18:43:26 on dist-test-slave-9jll"
server_key: "297a462578da96a7cc98ef864c367f02"
server_key_iv: "3b648b62426e3d7fe147f834f93f2c71"
server_key_version: "encryptionkey@0"
I20260927 18:43:26.111454 21250 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 18:43:26.115995 21264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:26.117805 21250 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:43:26.118077 21250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "dd6943365fc548a29df3f4b53a0b8f74"
format_stamp: "Formatted at 2026-09-27 18:43:26 on dist-test-slave-9jll"
server_key: "297a462578da96a7cc98ef864c367f02"
server_key_iv: "3b648b62426e3d7fe147f834f93f2c71"
server_key_version: "encryptionkey@0"
I20260927 18:43:26.118348 21250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:26.142434 21250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:26.143563 21250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:26.143884 21250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:26.165191 21250 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:34129
I20260927 18:43:26.165210 21315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:34129 every 8 connection(s)
I20260927 18:43:26.166914 21250 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:26.171599 21316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:26.176074 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 21250
I20260927 18:43:26.176364 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:26.177033 2429 external_mini_cluster.cc:1428] Setting key 03506c0f52f0bc8de6b2c5ac661c5528
I20260927 18:43:26.186813 21316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74: Bootstrap starting.
I20260927 18:43:26.190764 21316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:26.192315 21316 log.cc:824] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:26.193106 21319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:26.179514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.2.95.126:34129)
0927 18:43:26.180698 (+ 1184us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:26.180712 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:26.180765 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:43:26.182859 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:26.182892 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:26.183506 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:26.183960 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:26.184648 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:26.186090 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:26.187653 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:26.188403 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:26.188607 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:26.189884 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:26.189893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:26.189910 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:26.190575 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:26.190694 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:26.190719 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:26.190834 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:26.191109 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 18:43:26.191559 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":384,"threads_started":1}
I20260927 18:43:26.195986 21316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74: No bootstrap required, opened a new log
I20260927 18:43:26.204499 21316 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6943365fc548a29df3f4b53a0b8f74" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 34129 } }
I20260927 18:43:26.204995 21316 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:26.205103 21316 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6943365fc548a29df3f4b53a0b8f74, State: Initialized, Role: FOLLOWER
I20260927 18:43:26.205932 21316 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6943365fc548a29df3f4b53a0b8f74" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 34129 } }
I20260927 18:43:26.206243 21316 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:26.206403 21316 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:26.206583 21316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:26.209057 21316 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6943365fc548a29df3f4b53a0b8f74" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 34129 } }
I20260927 18:43:26.209756 21316 leader_election.cc:302] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd6943365fc548a29df3f4b53a0b8f74; no voters:
I20260927 18:43:26.210457 21316 leader_election.cc:288] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:26.210618 21321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:26.211521 21321 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [term 1 LEADER]: Becoming Leader. State: Replica: dd6943365fc548a29df3f4b53a0b8f74, State: Running, Role: LEADER
I20260927 18:43:26.212126 21321 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6943365fc548a29df3f4b53a0b8f74" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 34129 } }
I20260927 18:43:26.212980 21316 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:26.215114 21323 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd6943365fc548a29df3f4b53a0b8f74. Latest consensus state: current_term: 1 leader_uuid: "dd6943365fc548a29df3f4b53a0b8f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6943365fc548a29df3f4b53a0b8f74" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 34129 } } }
I20260927 18:43:26.216141 21323 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:26.215759 21322 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd6943365fc548a29df3f4b53a0b8f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6943365fc548a29df3f4b53a0b8f74" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 34129 } } }
I20260927 18:43:26.216881 21322 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:26.218411 21328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:26.224707 21328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:26.235515 21328 catalog_manager.cc:1347] Generated new cluster ID: 465791b3d3ed4c63b5b8a4b20552e185
I20260927 18:43:26.235675 21328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:26.264346 21328 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:26.266534 21328 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:26.277737 21328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74: Generated new TSK 0
I20260927 18:43:26.279124 21328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:26.293499 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:34129
--builtin_ntp_servers=127.2.95.84:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:43:26.699133 21340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:26.699536 21340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:26.699680 21340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:26.707856 21340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:26.708003 21340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:26.708156 21340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:43:26.719274 21340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:34129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:26.721014 21340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:26.723352 21340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:26.739005 21348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:26.739060 21346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:26.739060 21345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:26.740243 21340 server_base.cc:1034] running on GCE node
I20260927 18:43:26.740998 21340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:26.742270 21340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:26.743566 21340 hybrid_clock.cc:648] HybridClock initialized: now 1790534606743500 us; error 47 us; skew 500 ppm
I20260927 18:43:26.743975 21340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:43:26.747107 21340 webserver.cc:460] Webserver started at http://127.2.95.65:38961/ using document root <none> and password file <none>
I20260927 18:43:26.748013 21340 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:26.748118 21340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:26.748457 21340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:26.752127 21340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "a94727ed24624e80ac3b954554266460"
format_stamp: "Formatted at 2026-09-27 18:43:26 on dist-test-slave-9jll"
server_key: "d2c465dc5b5813a06f2dda29a7841cd3"
server_key_iv: "d739fd420f9af1b130f9b92a866707f4"
server_key_version: "encryptionkey@0"
I20260927 18:43:26.753079 21340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "a94727ed24624e80ac3b954554266460"
format_stamp: "Formatted at 2026-09-27 18:43:26 on dist-test-slave-9jll"
server_key: "d2c465dc5b5813a06f2dda29a7841cd3"
server_key_iv: "d739fd420f9af1b130f9b92a866707f4"
server_key_version: "encryptionkey@0"
I20260927 18:43:26.759564 21340 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 18:43:26.764012 21354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:26.765949 21340 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:43:26.766160 21340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "a94727ed24624e80ac3b954554266460"
format_stamp: "Formatted at 2026-09-27 18:43:26 on dist-test-slave-9jll"
server_key: "d2c465dc5b5813a06f2dda29a7841cd3"
server_key_iv: "d739fd420f9af1b130f9b92a866707f4"
server_key_version: "encryptionkey@0"
I20260927 18:43:26.766624 21340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:26.790479 21340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:26.791635 21340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:26.791960 21340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:26.793169 21340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:26.795205 21340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:26.795321 21340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:26.795457 21340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:26.795538 21340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:26.830209 21340 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:33943
I20260927 18:43:26.830276 21466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:33943 every 8 connection(s)
I20260927 18:43:26.831904 21340 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:43:26.837699 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 21340
I20260927 18:43:26.837924 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:43:26.838550 2429 external_mini_cluster.cc:1428] Setting key f8ee4ff67172398a4507f0038dae36f9
I20260927 18:43:26.842291 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:34129
--builtin_ntp_servers=127.2.95.84:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:43:26.848539 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:26.835361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:41573 (local address 127.2.95.126:34129)
0927 18:43:26.835979 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:26.835984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:26.837239 (+ 1255us) server_negotiation.cc:407] Connection header received
0927 18:43:26.839948 (+ 2709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:26.839954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:26.840070 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:26.840199 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:26.841954 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:26.842812 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:26.845001 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:26.845484 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:26.845614 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:26.847388 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:26.847394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:26.847397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:26.847427 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:26.847478 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:26.847493 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:26.847575 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:26.847965 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 18:43:26.848111 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":303,"threads_started":1}
I20260927 18:43:26.849980 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:26.835946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:34129 (local address 127.2.95.65:41573)
0927 18:43:26.837032 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:26.837074 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:26.839552 (+ 2478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:26.840454 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:26.840464 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:26.840592 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:26.841634 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:26.841731 (+ 97us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:26.842999 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:26.843004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:26.844795 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:26.844814 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:26.845799 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:26.845809 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:26.846085 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:26.846915 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:26.846918 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:26.846978 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:26.847105 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:26.847591 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:26.847605 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:26.847613 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 18:43:26.847923 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 18:43:26.848484 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":326,"threads_started":1}
I20260927 18:43:26.852141 21467 heartbeater.cc:347] Connected to a master server at 127.2.95.126:34129
I20260927 18:43:26.852548 21467 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:26.853746 21467 heartbeater.cc:511] Master 127.2.95.126:34129 requested a full tablet report, sending...
I20260927 18:43:26.856834 21281 ts_manager.cc:194] Registered new tserver with Master: a94727ed24624e80ac3b954554266460 (127.2.95.65:33943)
I20260927 18:43:26.859054 21281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.95.65:41573
W20260927 18:43:27.240442 21471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:27.240823 21471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:27.241037 21471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:27.249454 21471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:27.249615 21471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:27.249804 21471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:43:27.260857 21471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:34129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:27.262643 21471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:27.265035 21471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:27.280123 21476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:27.280161 21477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:27.280084 21479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:27.281054 21471 server_base.cc:1034] running on GCE node
I20260927 18:43:27.281847 21471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:27.283051 21471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:27.284291 21471 hybrid_clock.cc:648] HybridClock initialized: now 1790534607284221 us; error 63 us; skew 500 ppm
I20260927 18:43:27.284742 21471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:43:27.287569 21471 webserver.cc:460] Webserver started at http://127.2.95.66:42489/ using document root <none> and password file <none>
I20260927 18:43:27.288420 21471 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:27.288512 21471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:27.288823 21471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:27.292212 21471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "4a14e0e2def040f18a89a22024258ee8"
format_stamp: "Formatted at 2026-09-27 18:43:27 on dist-test-slave-9jll"
server_key: "d85259afec52dd575d4e7fc86d110dce"
server_key_iv: "9c560c5d48b377e2f5029efe1f2a4f68"
server_key_version: "encryptionkey@0"
I20260927 18:43:27.293064 21471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "4a14e0e2def040f18a89a22024258ee8"
format_stamp: "Formatted at 2026-09-27 18:43:27 on dist-test-slave-9jll"
server_key: "d85259afec52dd575d4e7fc86d110dce"
server_key_iv: "9c560c5d48b377e2f5029efe1f2a4f68"
server_key_version: "encryptionkey@0"
I20260927 18:43:27.298928 21471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 18:43:27.303231 21485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:27.305060 21471 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:43:27.305246 21471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "4a14e0e2def040f18a89a22024258ee8"
format_stamp: "Formatted at 2026-09-27 18:43:27 on dist-test-slave-9jll"
server_key: "d85259afec52dd575d4e7fc86d110dce"
server_key_iv: "9c560c5d48b377e2f5029efe1f2a4f68"
server_key_version: "encryptionkey@0"
I20260927 18:43:27.305498 21471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:27.331121 21471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:27.332226 21471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:27.332540 21471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:27.333791 21471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:27.335747 21471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:27.335845 21471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:27.335945 21471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:27.336002 21471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:27.369796 21471 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:41241
I20260927 18:43:27.369863 21597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:41241 every 8 connection(s)
I20260927 18:43:27.371616 21471 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:43:27.379498 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 21471
I20260927 18:43:27.379734 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:43:27.380249 2429 external_mini_cluster.cc:1428] Setting key f2787385c678f77d776455e2473b27e4
I20260927 18:43:27.384016 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:34129
--builtin_ntp_servers=127.2.95.84:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:43:27.388842 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:27.375152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:41441 (local address 127.2.95.126:34129)
0927 18:43:27.375751 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:27.375756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:27.377130 (+ 1374us) server_negotiation.cc:407] Connection header received
0927 18:43:27.379321 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:27.379328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:27.379446 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:27.379590 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:27.381031 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.382204 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.384620 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.385436 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.385620 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:27.387840 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:27.387847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:27.387851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:27.387879 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:27.387939 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:27.387954 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:27.388042 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:27.388393 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 18:43:27.388511 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":268,"threads_started":1}
I20260927 18:43:27.390563 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:27.375821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:34129 (local address 127.2.95.66:41441)
0927 18:43:27.376941 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:27.376990 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:27.378929 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:27.379945 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:27.379955 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:27.380077 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:27.380824 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:27.380845 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.382384 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.382390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:27.384426 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:27.384443 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.385833 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.385863 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:27.386260 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:27.387241 (+ 981us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:27.387245 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:27.387389 (+ 144us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:27.387544 (+ 155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:27.388058 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:27.388069 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:27.388078 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:43:27.388291 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 18:43:27.388837 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":338,"threads_started":1}
I20260927 18:43:27.392354 21598 heartbeater.cc:347] Connected to a master server at 127.2.95.126:34129
I20260927 18:43:27.392745 21598 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:27.393860 21598 heartbeater.cc:511] Master 127.2.95.126:34129 requested a full tablet report, sending...
I20260927 18:43:27.395982 21281 ts_manager.cc:194] Registered new tserver with Master: 4a14e0e2def040f18a89a22024258ee8 (127.2.95.66:41241)
I20260927 18:43:27.397240 21281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.95.66:41441
W20260927 18:43:27.768035 21602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:27.768409 21602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:27.768503 21602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:27.776669 21602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:27.776800 21602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:27.776928 21602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:43:27.786737 21602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:34129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:27.788372 21602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:27.790565 21602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:27.806375 21607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:27.806421 21610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:27.806452 21608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:27.807461 21602 server_base.cc:1034] running on GCE node
I20260927 18:43:27.808353 21602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:27.809643 21602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:27.810937 21602 hybrid_clock.cc:648] HybridClock initialized: now 1790534607810867 us; error 66 us; skew 500 ppm
I20260927 18:43:27.811324 21602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:43:27.821394 21602 webserver.cc:460] Webserver started at http://127.2.95.67:45011/ using document root <none> and password file <none>
I20260927 18:43:27.822389 21602 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:27.822537 21602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:27.822930 21602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:27.826772 21602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "d65216532718442ab7367dbe2f7509f3"
format_stamp: "Formatted at 2026-09-27 18:43:27 on dist-test-slave-9jll"
server_key: "48f1054b5402eb830850758a7029e138"
server_key_iv: "9d80dc31db076791bd958448e15da215"
server_key_version: "encryptionkey@0"
I20260927 18:43:27.827795 21602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "d65216532718442ab7367dbe2f7509f3"
format_stamp: "Formatted at 2026-09-27 18:43:27 on dist-test-slave-9jll"
server_key: "48f1054b5402eb830850758a7029e138"
server_key_iv: "9d80dc31db076791bd958448e15da215"
server_key_version: "encryptionkey@0"
I20260927 18:43:27.834465 21602 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 18:43:27.838845 21616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:27.840462 21602 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260927 18:43:27.840633 21602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "d65216532718442ab7367dbe2f7509f3"
format_stamp: "Formatted at 2026-09-27 18:43:27 on dist-test-slave-9jll"
server_key: "48f1054b5402eb830850758a7029e138"
server_key_iv: "9d80dc31db076791bd958448e15da215"
server_key_version: "encryptionkey@0"
I20260927 18:43:27.840855 21602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:27.859215 21602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:27.860258 21602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:27.860543 21602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:27.861683 21602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:27.863446 21602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:27.863544 21602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:27.863586 21467 heartbeater.cc:503] Master 127.2.95.126:34129 was elected leader, sending a full tablet report...
I20260927 18:43:27.863656 21602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:27.863721 21602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:27.899003 21602 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:44147
I20260927 18:43:27.899055 21728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:44147 every 8 connection(s)
I20260927 18:43:27.900633 21602 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:43:27.909510 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 21602
I20260927 18:43:27.909844 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:43:27.910441 2429 external_mini_cluster.cc:1428] Setting key 62db2f617e28c1a9227a5fa05a03cb12
I20260927 18:43:27.917686 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:27.903667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:57137 (local address 127.2.95.126:34129)
0927 18:43:27.904233 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:27.904238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:27.905940 (+ 1702us) server_negotiation.cc:407] Connection header received
0927 18:43:27.908986 (+ 3046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:27.908994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:27.909117 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:27.909250 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:27.910892 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.912018 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.914410 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.914933 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.915067 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:27.916665 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:27.916670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:27.916674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:27.916697 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:27.916746 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:27.916759 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:27.916828 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:27.917080 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 18:43:27.917183 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":237,"threads_started":1}
I20260927 18:43:27.919066 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:27.904328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:34129 (local address 127.2.95.67:57137)
0927 18:43:27.905730 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:27.905786 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:27.908552 (+ 2766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:27.909578 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:27.909589 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:27.909776 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:27.910671 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:27.910699 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.912220 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.912226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:27.914237 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:27.914257 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.915173 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.915179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:27.915451 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:27.916293 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:27.916295 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:27.916350 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:27.916456 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:27.916875 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:27.916884 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:27.916891 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:43:27.917057 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 18:43:27.917517 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":438,"threads_started":1}
I20260927 18:43:27.920729 21729 heartbeater.cc:347] Connected to a master server at 127.2.95.126:34129
I20260927 18:43:27.921135 21729 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:27.922257 21729 heartbeater.cc:511] Master 127.2.95.126:34129 requested a full tablet report, sending...
I20260927 18:43:27.924085 21281 ts_manager.cc:194] Registered new tserver with Master: d65216532718442ab7367dbe2f7509f3 (127.2.95.67:44147)
I20260927 18:43:27.925222 21281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.95.67:57137
I20260927 18:43:27.927269 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:43:27.938779 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:27.932088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.2.95.126:34129)
0927 18:43:27.932343 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:27.932351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:27.932630 (+ 279us) server_negotiation.cc:407] Connection header received
0927 18:43:27.932785 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:27.932789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:27.932864 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:27.932954 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:27.933463 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.934283 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.935970 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.936454 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.936568 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:27.937929 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:27.937935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:27.937938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:27.937962 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:27.938011 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:27.938025 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:27.938158 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:27.938362 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 18:43:27.938446 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 18:43:27.943373 21281 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:43:27.951099 21281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:43:27.977209 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:27.965333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:41241 (local address 127.0.0.1:45274)
0927 18:43:27.966230 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:27.966272 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:27.966469 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:27.967904 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:27.967913 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:27.967950 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:27.968497 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:27.968513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.971014 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.971020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:27.973728 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:27.973746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.976004 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.976012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:27.976269 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:27.976344 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 18:43:27.976478 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 18:43:27.976747 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":352,"threads_started":1}
I20260927 18:43:27.978999 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:27.965597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.2.95.66:41241)
0927 18:43:27.967124 (+ 1527us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:27.967136 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:27.967175 (+ 39us) server_negotiation.cc:407] Connection header received
0927 18:43:27.967298 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:27.967308 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:27.967625 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:27.968759 (+ 1134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:27.969080 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.970839 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.974035 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.975738 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.977841 (+ 2103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:27.978223 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:27.978314 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 18:43:27.978553 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":1176,"threads_started":1}
I20260927 18:43:27.980472 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:27.966649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:44147 (local address 127.0.0.1:54534)
0927 18:43:27.970042 (+ 3393us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:27.970059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:27.970183 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:27.971298 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:27.971303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:27.971332 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:27.972548 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:27.972563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.974769 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.974776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:27.977105 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:27.977119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.979733 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.979741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:27.979945 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:27.979979 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 18:43:27.980060 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:43:27.980211 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3287,"thread_start_us":3178,"threads_started":1}
I20260927 18:43:27.981310 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:27.967091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.2.95.67:44147)
0927 18:43:27.968000 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:27.968019 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:27.970321 (+ 2302us) server_negotiation.cc:407] Connection header received
0927 18:43:27.970470 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:27.970478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:27.970760 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:27.970974 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:27.972765 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.974589 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.977366 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.979077 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.979229 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:27.979572 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:27.980719 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0927 18:43:27.980908 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":585,"threads_started":1}
I20260927 18:43:27.984594 21533 tablet_service.cc:1467] Processing CreateTablet for tablet 48bf84de3c3d48e7a298ba9fcdf5acc0 (DEFAULT_TABLE table=test-table [id=7a74f2ceda764554ba9d9b14dceaa82f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:27.985001 21664 tablet_service.cc:1467] Processing CreateTablet for tablet 48bf84de3c3d48e7a298ba9fcdf5acc0 (DEFAULT_TABLE table=test-table [id=7a74f2ceda764554ba9d9b14dceaa82f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:27.986275 21533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bf84de3c3d48e7a298ba9fcdf5acc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:27.986406 21664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bf84de3c3d48e7a298ba9fcdf5acc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:27.988197 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:27.970190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:33943 (local address 127.0.0.1:51642)
0927 18:43:27.977507 (+ 7317us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:27.977527 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:27.977627 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:27.980950 (+ 3323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:27.980954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:27.980984 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:27.981161 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:27.981171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.983555 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.983561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:27.985424 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:27.985440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.987309 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.987315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:27.987515 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:27.987551 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 18:43:27.987666 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 18:43:27.987839 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7218,"thread_start_us":9316,"threads_started":1}
I20260927 18:43:27.988821 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:27.970638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.2.95.65:33943)
0927 18:43:27.980229 (+ 9591us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:27.980240 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:27.980275 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:43:27.980394 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:27.980403 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:27.980717 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:27.980997 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:27.981755 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.983393 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.985605 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:27.987113 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:27.987275 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:27.987588 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:27.988243 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 18:43:27.988443 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9370,"thread_start_us":9205,"threads_started":1}
I20260927 18:43:27.993966 21402 tablet_service.cc:1467] Processing CreateTablet for tablet 48bf84de3c3d48e7a298ba9fcdf5acc0 (DEFAULT_TABLE table=test-table [id=7a74f2ceda764554ba9d9b14dceaa82f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:27.995990 21402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bf84de3c3d48e7a298ba9fcdf5acc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:28.000914 21748 tablet_bootstrap.cc:492] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8: Bootstrap starting.
I20260927 18:43:28.003571 21749 tablet_bootstrap.cc:492] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3: Bootstrap starting.
I20260927 18:43:28.005973 21748 tablet_bootstrap.cc:654] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:28.008085 21749 tablet_bootstrap.cc:654] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:28.008172 21748 log.cc:824] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:28.009603 21749 log.cc:824] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:28.012348 21748 tablet_bootstrap.cc:492] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8: No bootstrap required, opened a new log
I20260927 18:43:28.012734 21748 ts_tablet_manager.cc:1397] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260927 18:43:28.013183 21749 tablet_bootstrap.cc:492] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3: No bootstrap required, opened a new log
I20260927 18:43:28.013518 21749 ts_tablet_manager.cc:1397] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 18:43:28.016039 21752 tablet_bootstrap.cc:492] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460: Bootstrap starting.
I20260927 18:43:28.021340 21752 tablet_bootstrap.cc:654] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:28.023398 21752 log.cc:824] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:28.023170 21749 raft_consensus.cc:348] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a14e0e2def040f18a89a22024258ee8" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41241 } } peers { permanent_uuid: "d65216532718442ab7367dbe2f7509f3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44147 } } peers { permanent_uuid: "a94727ed24624e80ac3b954554266460" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33943 } }
I20260927 18:43:28.023886 21749 raft_consensus.cc:374] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:28.023999 21749 raft_consensus.cc:729] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d65216532718442ab7367dbe2f7509f3, State: Initialized, Role: FOLLOWER
I20260927 18:43:28.024869 21749 consensus_queue.cc:260] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a14e0e2def040f18a89a22024258ee8" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41241 } } peers { permanent_uuid: "d65216532718442ab7367dbe2f7509f3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44147 } } peers { permanent_uuid: "a94727ed24624e80ac3b954554266460" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33943 } }
I20260927 18:43:28.024762 21748 raft_consensus.cc:348] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a14e0e2def040f18a89a22024258ee8" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41241 } } peers { permanent_uuid: "d65216532718442ab7367dbe2f7509f3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44147 } } peers { permanent_uuid: "a94727ed24624e80ac3b954554266460" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33943 } }
I20260927 18:43:28.025369 21748 raft_consensus.cc:374] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:28.025478 21748 raft_consensus.cc:729] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a14e0e2def040f18a89a22024258ee8, State: Initialized, Role: FOLLOWER
I20260927 18:43:28.026342 21729 heartbeater.cc:503] Master 127.2.95.126:34129 was elected leader, sending a full tablet report...
I20260927 18:43:28.026275 21748 consensus_queue.cc:260] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a14e0e2def040f18a89a22024258ee8" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41241 } } peers { permanent_uuid: "d65216532718442ab7367dbe2f7509f3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44147 } } peers { permanent_uuid: "a94727ed24624e80ac3b954554266460" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33943 } }
I20260927 18:43:28.027544 21598 heartbeater.cc:503] Master 127.2.95.126:34129 was elected leader, sending a full tablet report...
I20260927 18:43:28.028930 21749 ts_tablet_manager.cc:1428] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 18:43:28.030309 21748 ts_tablet_manager.cc:1428] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260927 18:43:28.034354 21752 tablet_bootstrap.cc:492] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460: No bootstrap required, opened a new log
I20260927 18:43:28.034751 21752 ts_tablet_manager.cc:1397] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20260927 18:43:28.043463 21752 raft_consensus.cc:348] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a14e0e2def040f18a89a22024258ee8" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41241 } } peers { permanent_uuid: "d65216532718442ab7367dbe2f7509f3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44147 } } peers { permanent_uuid: "a94727ed24624e80ac3b954554266460" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33943 } }
I20260927 18:43:28.043977 21752 raft_consensus.cc:374] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:28.044097 21752 raft_consensus.cc:729] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94727ed24624e80ac3b954554266460, State: Initialized, Role: FOLLOWER
I20260927 18:43:28.044876 21752 consensus_queue.cc:260] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a14e0e2def040f18a89a22024258ee8" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41241 } } peers { permanent_uuid: "d65216532718442ab7367dbe2f7509f3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44147 } } peers { permanent_uuid: "a94727ed24624e80ac3b954554266460" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33943 } }
I20260927 18:43:28.047102 21752 ts_tablet_manager.cc:1428] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260927 18:43:28.084895 21468 tablet.cc:2369] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:43:28.124459 21599 tablet.cc:2369] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:43:28.152611 21730 tablet.cc:2369] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:43:28.201933 21754 raft_consensus.cc:482] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:43:28.202304 21754 raft_consensus.cc:504] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a14e0e2def040f18a89a22024258ee8" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41241 } } peers { permanent_uuid: "d65216532718442ab7367dbe2f7509f3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44147 } } peers { permanent_uuid: "a94727ed24624e80ac3b954554266460" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33943 } }
I20260927 18:43:28.204432 21754 leader_election.cc:288] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a14e0e2def040f18a89a22024258ee8 (127.2.95.66:41241), a94727ed24624e80ac3b954554266460 (127.2.95.65:33943)
I20260927 18:43:28.211184 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:28.204742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:41241 (local address 127.2.95.67:37829)
0927 18:43:28.204947 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:28.204970 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:28.205123 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:28.205792 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:28.205797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:28.205823 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:28.206025 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:28.206037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.207361 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.207375 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:28.209043 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:28.209056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.210456 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.210462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:28.210639 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:28.210684 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 18:43:28.210776 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 18:43:28.210915 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 18:43:28.211256 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:28.204974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:37829 (local address 127.2.95.66:41241)
0927 18:43:28.205186 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:28.205194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:28.205218 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:43:28.205361 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:28.205371 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:28.205483 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:28.205600 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:28.206216 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.207224 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.209194 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.210297 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.210443 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:28.210547 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:28.210815 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 18:43:28.210964 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:43:28.212252 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:28.205421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:34827 (local address 127.2.95.65:33943)
0927 18:43:28.205649 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:28.205655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:28.206000 (+ 345us) server_negotiation.cc:407] Connection header received
0927 18:43:28.206146 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:28.206153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:28.206247 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:28.206316 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:28.206947 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.207815 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.210118 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.211299 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.211442 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:28.211544 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:28.211947 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 18:43:28.212058 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 18:43:28.212324 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:28.205146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:33943 (local address 127.2.95.67:34827)
0927 18:43:28.205889 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:28.205904 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:28.206017 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:28.206353 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:28.206357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:28.206381 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:28.206788 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:28.206800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.208001 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.208005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:28.209900 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:28.209914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.211658 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.211665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:28.211795 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:28.211819 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 18:43:28.211898 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:43:28.212022 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":354,"threads_started":1}
I20260927 18:43:28.212503 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bf84de3c3d48e7a298ba9fcdf5acc0" candidate_uuid: "d65216532718442ab7367dbe2f7509f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a14e0e2def040f18a89a22024258ee8" is_pre_election: true
I20260927 18:43:28.213212 21553 raft_consensus.cc:2454] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d65216532718442ab7367dbe2f7509f3 in term 0.
I20260927 18:43:28.213438 21422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bf84de3c3d48e7a298ba9fcdf5acc0" candidate_uuid: "d65216532718442ab7367dbe2f7509f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94727ed24624e80ac3b954554266460" is_pre_election: true
I20260927 18:43:28.214061 21422 raft_consensus.cc:2454] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d65216532718442ab7367dbe2f7509f3 in term 0.
I20260927 18:43:28.214025 21619 leader_election.cc:302] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a14e0e2def040f18a89a22024258ee8, d65216532718442ab7367dbe2f7509f3; no voters:
I20260927 18:43:28.214533 21754 raft_consensus.cc:2781] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:43:28.214751 21754 raft_consensus.cc:482] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:43:28.214875 21754 raft_consensus.cc:3037] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:28.217437 21754 raft_consensus.cc:504] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a14e0e2def040f18a89a22024258ee8" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41241 } } peers { permanent_uuid: "d65216532718442ab7367dbe2f7509f3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44147 } } peers { permanent_uuid: "a94727ed24624e80ac3b954554266460" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33943 } }
I20260927 18:43:28.218199 21754 leader_election.cc:288] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [CANDIDATE]: Term 1 election: Requested vote from peers 4a14e0e2def040f18a89a22024258ee8 (127.2.95.66:41241), a94727ed24624e80ac3b954554266460 (127.2.95.65:33943)
I20260927 18:43:28.218690 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bf84de3c3d48e7a298ba9fcdf5acc0" candidate_uuid: "d65216532718442ab7367dbe2f7509f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a14e0e2def040f18a89a22024258ee8"
I20260927 18:43:28.218729 21422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bf84de3c3d48e7a298ba9fcdf5acc0" candidate_uuid: "d65216532718442ab7367dbe2f7509f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94727ed24624e80ac3b954554266460"
I20260927 18:43:28.218942 21422 raft_consensus.cc:3037] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:28.218969 21553 raft_consensus.cc:3037] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:28.222489 21553 raft_consensus.cc:2454] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d65216532718442ab7367dbe2f7509f3 in term 1.
I20260927 18:43:28.222489 21422 raft_consensus.cc:2454] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d65216532718442ab7367dbe2f7509f3 in term 1.
I20260927 18:43:28.222918 21619 leader_election.cc:302] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a94727ed24624e80ac3b954554266460, d65216532718442ab7367dbe2f7509f3; no voters:
I20260927 18:43:28.223285 21754 raft_consensus.cc:2781] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:28.223804 21754 raft_consensus.cc:686] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 1 LEADER]: Becoming Leader. State: Replica: d65216532718442ab7367dbe2f7509f3, State: Running, Role: LEADER
I20260927 18:43:28.224411 21754 consensus_queue.cc:237] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a14e0e2def040f18a89a22024258ee8" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41241 } } peers { permanent_uuid: "d65216532718442ab7367dbe2f7509f3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44147 } } peers { permanent_uuid: "a94727ed24624e80ac3b954554266460" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33943 } }
I20260927 18:43:28.230017 21281 catalog_manager.cc:5495] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d65216532718442ab7367dbe2f7509f3 (127.2.95.67). New cstate: current_term: 1 leader_uuid: "d65216532718442ab7367dbe2f7509f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a14e0e2def040f18a89a22024258ee8" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d65216532718442ab7367dbe2f7509f3" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a94727ed24624e80ac3b954554266460" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33943 } health_report { overall_health: UNKNOWN } } }
I20260927 18:43:28.282732 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:28.276616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.2.95.67:44147)
0927 18:43:28.276901 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:28.276906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:28.276928 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:43:28.277028 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:28.277032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:28.277131 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:28.277280 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:43:28.277740 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.278552 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.280077 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.280544 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.280658 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:28.281273 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:43:28.282057 (+ 784us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:43:28.282257 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:28.282399 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 18:43:28.282452 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260927 18:43:28.292892 21422 raft_consensus.cc:1261] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460 [term 1 FOLLOWER]: Refusing update from remote peer d65216532718442ab7367dbe2f7509f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:28.292894 21553 raft_consensus.cc:1261] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8 [term 1 FOLLOWER]: Refusing update from remote peer d65216532718442ab7367dbe2f7509f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:28.294065 21758 consensus_queue.cc:1035] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a94727ed24624e80ac3b954554266460" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 33943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:28.294468 21754 consensus_queue.cc:1035] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a14e0e2def040f18a89a22024258ee8" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 41241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:28.325263 21280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:28.325654 21280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:28.330583 21280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd6943365fc548a29df3f4b53a0b8f74: Sending DeleteTablet for 3 replicas of tablet 48bf84de3c3d48e7a298ba9fcdf5acc0
I20260927 18:43:28.332067 21533 tablet_service.cc:1514] Processing DeleteTablet for tablet 48bf84de3c3d48e7a298ba9fcdf5acc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:28 UTC) from {username='slave'} at 127.0.0.1:45274
I20260927 18:43:28.332108 21664 tablet_service.cc:1514] Processing DeleteTablet for tablet 48bf84de3c3d48e7a298ba9fcdf5acc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:28 UTC) from {username='slave'} at 127.0.0.1:54534
I20260927 18:43:28.333005 21402 tablet_service.cc:1514] Processing DeleteTablet for tablet 48bf84de3c3d48e7a298ba9fcdf5acc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:28 UTC) from {username='slave'} at 127.0.0.1:51642
I20260927 18:43:28.338474 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 21340
I20260927 18:43:28.342273 21773 tablet_replica.cc:330] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460: stopping tablet replica
I20260927 18:43:28.342945 21773 raft_consensus.cc:2229] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:28.343798 21773 raft_consensus.cc:2258] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:28.346832 21773 ts_tablet_manager.cc:1905] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:28.355312 21773 ts_tablet_manager.cc:1918] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:28.355654 21773 log.cc:1196] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/48bf84de3c3d48e7a298ba9fcdf5acc0
I20260927 18:43:28.356350 21773 ts_tablet_manager.cc:1939] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P a94727ed24624e80ac3b954554266460: Deleting consensus metadata
I20260927 18:43:28.359194 21267 catalog_manager.cc:4841] TS a94727ed24624e80ac3b954554266460 (127.2.95.65:33943): tablet 48bf84de3c3d48e7a298ba9fcdf5acc0 (table test-table [id=7a74f2ceda764554ba9d9b14dceaa82f]) successfully deleted
I20260927 18:43:28.362358 21771 tablet_replica.cc:330] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8: stopping tablet replica
I20260927 18:43:28.363004 21771 raft_consensus.cc:2229] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:28.363744 21771 raft_consensus.cc:2258] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:28.364797 21772 tablet_replica.cc:330] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3: stopping tablet replica
I20260927 18:43:28.365379 21772 raft_consensus.cc:2229] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:43:28.366302 21772 raft_consensus.cc:2258] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:28.368439 21771 ts_tablet_manager.cc:1905] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:28.371260 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:28.341772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.2.95.65:33943)
0927 18:43:28.342084 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:28.342092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:28.362114 (+ 20022us) server_negotiation.cc:407] Connection header received
0927 18:43:28.362644 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:28.362649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:28.362752 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:28.362830 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:28.363695 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.364808 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.366665 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.367263 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.367386 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:28.369166 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:28.369179 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:28.369200 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:28.370163 (+ 963us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:28.370237 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:28.370265 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:28.370369 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:28.370655 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 18:43:28.370813 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260927 18:43:28.371584 21772 ts_tablet_manager.cc:1905] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:28.372591 21462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:28.381594 21771 ts_tablet_manager.cc:1918] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:28.381992 21771 log.cc:1196] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/48bf84de3c3d48e7a298ba9fcdf5acc0
I20260927 18:43:28.382026 21772 ts_tablet_manager.cc:1918] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:28.382217 21772 log.cc:1196] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/48bf84de3c3d48e7a298ba9fcdf5acc0
I20260927 18:43:28.382786 21772 ts_tablet_manager.cc:1939] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P d65216532718442ab7367dbe2f7509f3: Deleting consensus metadata
I20260927 18:43:28.382799 21771 ts_tablet_manager.cc:1939] T 48bf84de3c3d48e7a298ba9fcdf5acc0 P 4a14e0e2def040f18a89a22024258ee8: Deleting consensus metadata
I20260927 18:43:28.384999 21267 catalog_manager.cc:4841] TS 4a14e0e2def040f18a89a22024258ee8 (127.2.95.66:41241): tablet 48bf84de3c3d48e7a298ba9fcdf5acc0 (table test-table [id=7a74f2ceda764554ba9d9b14dceaa82f]) successfully deleted
I20260927 18:43:28.385350 21267 catalog_manager.cc:4841] TS d65216532718442ab7367dbe2f7509f3 (127.2.95.67:44147): tablet 48bf84de3c3d48e7a298ba9fcdf5acc0 (table test-table [id=7a74f2ceda764554ba9d9b14dceaa82f]) successfully deleted
I20260927 18:43:28.472836 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 21340
I20260927 18:43:28.490549 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 21471
I20260927 18:43:28.499177 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:28.491514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.2.95.66:41241)
0927 18:43:28.491741 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:28.491748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:28.491773 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:43:28.491941 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:28.491947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:28.492093 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:28.492221 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:28.492613 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.493870 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.495250 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.495850 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.495981 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:28.497110 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:28.497123 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:28.497149 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:28.498275 (+ 1126us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:28.498351 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:28.498389 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:28.498513 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:28.498624 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 18:43:28.498759 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:43:28.500072 21593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:28.595854 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 21471
I20260927 18:43:28.620146 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 21602
I20260927 18:43:28.627983 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:28.621059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54546 (local address 127.2.95.67:44147)
0927 18:43:28.621314 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:28.621325 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:28.621348 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:43:28.621510 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:28.621517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:28.621623 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:28.621753 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:28.622183 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.623062 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.624416 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:28.624882 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:28.624997 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:28.626333 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:28.626342 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:28.626367 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:28.627134 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:28.627192 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:28.627221 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:28.627320 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:28.627476 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 18:43:28.627600 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 18:43:28.628814 21724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:28.761767 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 21602
I20260927 18:43:28.781116 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 21250
I20260927 18:43:28.782517 21311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:28.906531 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 21250
2026-09-27T18:43:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3351 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:29 dist-test-slave-9jll krb5kdc[21781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:29 dist-test-slave-9jll krb5kdc[21781](info): set up 2 sockets
Sep 27 18:43:29 dist-test-slave-9jll krb5kdc[21781](info): commencing operation
krb5kdc: starting...
W20260927 18:43:29.966256 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.009s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:30 dist-test-slave-9jll krb5kdc[21781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:43:30.270269 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:45557
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:45557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:30.667809 21794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:30.668293 21794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:30.668463 21794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:30.676963 21794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:30.677141 21794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:30.677210 21794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:30.677309 21794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:30.677384 21794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:30.689188 21794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:45557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:45557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:30.691167 21794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:30.692901 21794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:30.707518 21799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:30.707453 21800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:30.707479 21802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:30.708122 21794 server_base.cc:1034] running on GCE node
I20260927 18:43:30.709175 21794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:30.711112 21794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:30.712370 21794 hybrid_clock.cc:648] HybridClock initialized: now 1790534610712309 us; error 35 us; skew 500 ppm
I20260927 18:43:30.712777 21794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:30 dist-test-slave-9jll krb5kdc[21781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534610, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:30.716233 21794 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:43:30.718110 21794 webserver.cc:460] Webserver started at http://127.2.95.126:36549/ using document root <none> and password file <none>
I20260927 18:43:30.718946 21794 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:30.719069 21794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:30.719398 21794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:30.723282 21794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "e09393565190402ebee12c3bfa12b178"
format_stamp: "Formatted at 2026-09-27 18:43:30 on dist-test-slave-9jll"
server_key: "38155c9e2afdaf17a34a74e3c0875103"
server_key_iv: "7b7974fe64c75e716eb8b59deabf24eb"
server_key_version: "encryptionkey@0"
I20260927 18:43:30.724270 21794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "e09393565190402ebee12c3bfa12b178"
format_stamp: "Formatted at 2026-09-27 18:43:30 on dist-test-slave-9jll"
server_key: "38155c9e2afdaf17a34a74e3c0875103"
server_key_iv: "7b7974fe64c75e716eb8b59deabf24eb"
server_key_version: "encryptionkey@0"
I20260927 18:43:30.731079 21794 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 18:43:30.735417 21809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:30.737221 21794 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:43:30.737426 21794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "e09393565190402ebee12c3bfa12b178"
format_stamp: "Formatted at 2026-09-27 18:43:30 on dist-test-slave-9jll"
server_key: "38155c9e2afdaf17a34a74e3c0875103"
server_key_iv: "7b7974fe64c75e716eb8b59deabf24eb"
server_key_version: "encryptionkey@0"
I20260927 18:43:30.737687 21794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:30.782661 21794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:30.788215 21794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:30.788586 21794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:30.809324 21794 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:45557
I20260927 18:43:30.809324 21860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:45557 every 8 connection(s)
I20260927 18:43:30.811242 21794 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:30.812551 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 21794
I20260927 18:43:30.812757 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:30.813287 2429 external_mini_cluster.cc:1428] Setting key 123f76b400d7853d89605ec9eaad7b29
I20260927 18:43:30.816798 21861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:43:30 dist-test-slave-9jll krb5kdc[21781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:43:30.831626 21861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178: Bootstrap starting.
I20260927 18:43:30.836894 21861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:30.838403 21861 log.cc:824] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:30.841969 21861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178: No bootstrap required, opened a new log
I20260927 18:43:30.842000 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:30.815589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54864 (local address 127.2.95.126:45557)
0927 18:43:30.816479 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:30.816498 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:30.816558 (+ 60us) server_negotiation.cc:407] Connection header received
0927 18:43:30.819139 (+ 2581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:30.819173 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:30.819758 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:30.820293 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:30.822010 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:30.824288 (+ 2278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:30.825860 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:30.826929 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:30.827125 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:30.832083 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:30.832095 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:30.832121 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:30.832186 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:30.836688 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:30.837760 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:30.837805 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:30.837815 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:30.838109 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:30.838626 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:30.838632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:30.838635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:30.839459 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:30.839825 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:30.840322 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 18:43:30.840817 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":358,"threads_started":1}
I20260927 18:43:30.850543 21861 raft_consensus.cc:348] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09393565190402ebee12c3bfa12b178" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45557 } }
I20260927 18:43:30.850987 21861 raft_consensus.cc:374] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:30.851099 21861 raft_consensus.cc:729] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09393565190402ebee12c3bfa12b178, State: Initialized, Role: FOLLOWER
I20260927 18:43:30.851766 21861 consensus_queue.cc:260] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09393565190402ebee12c3bfa12b178" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45557 } }
I20260927 18:43:30.852018 21861 raft_consensus.cc:388] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:30.852160 21861 raft_consensus.cc:482] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:30.852342 21861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:30.854792 21861 raft_consensus.cc:504] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09393565190402ebee12c3bfa12b178" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45557 } }
I20260927 18:43:30.855391 21861 leader_election.cc:302] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e09393565190402ebee12c3bfa12b178; no voters:
I20260927 18:43:30.856086 21861 leader_election.cc:288] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:30.856215 21866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:30.857087 21866 raft_consensus.cc:686] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [term 1 LEADER]: Becoming Leader. State: Replica: e09393565190402ebee12c3bfa12b178, State: Running, Role: LEADER
I20260927 18:43:30.857709 21866 consensus_queue.cc:237] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09393565190402ebee12c3bfa12b178" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45557 } }
I20260927 18:43:30.858618 21861 sys_catalog.cc:564] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:30.861398 21868 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e09393565190402ebee12c3bfa12b178. Latest consensus state: current_term: 1 leader_uuid: "e09393565190402ebee12c3bfa12b178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09393565190402ebee12c3bfa12b178" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45557 } } }
I20260927 18:43:30.861425 21867 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e09393565190402ebee12c3bfa12b178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09393565190402ebee12c3bfa12b178" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45557 } } }
I20260927 18:43:30.861914 21868 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:30.862025 21867 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:30.867017 21874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:30.872700 21874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:30.882776 21874 catalog_manager.cc:1347] Generated new cluster ID: 130791da77ea4b8ba103531bef0ed442
I20260927 18:43:30.882929 21874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:30.911417 21874 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:30.913084 21874 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:30.925731 21874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178: Generated new TSK 0
I20260927 18:43:30.926968 21874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.95.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
Entry for principal kudu/127.2.95.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab.
I20260927 18:43:31.005847 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:45557
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:31.427546 21887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:31.427918 21887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:31.428050 21887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:31.436127 21887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:31.436276 21887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:31.436444 21887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:43:31.447463 21887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.65.keytab
--principal=kudu/127.2.95.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:45557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:31.449147 21887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:31.450858 21887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:31.466465 21892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:31.466521 21895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:31.466555 21893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:31.467788 21887 server_base.cc:1034] running on GCE node
I20260927 18:43:31.468540 21887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:31.469820 21887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:31.471113 21887 hybrid_clock.cc:648] HybridClock initialized: now 1790534611471044 us; error 56 us; skew 500 ppm
I20260927 18:43:31.471515 21887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:31 dist-test-slave-9jll krb5kdc[21781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534611, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:31.475312 21887 init.cc:377] Logged in from keytab as kudu/127.2.95.65@KRBTEST.COM (short username kudu)
I20260927 18:43:31.477185 21887 webserver.cc:460] Webserver started at http://127.2.95.65:42611/ using document root <none> and password file <none>
I20260927 18:43:31.478092 21887 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:31.478219 21887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:31.478569 21887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:31.482149 21887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "4bbc17c503cc452ab19404c94888f829"
format_stamp: "Formatted at 2026-09-27 18:43:31 on dist-test-slave-9jll"
server_key: "ae00e70f15fc454b63853aaca407c26f"
server_key_iv: "871720b753f451f86b03456a5a5b68b2"
server_key_version: "encryptionkey@0"
I20260927 18:43:31.483038 21887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "4bbc17c503cc452ab19404c94888f829"
format_stamp: "Formatted at 2026-09-27 18:43:31 on dist-test-slave-9jll"
server_key: "ae00e70f15fc454b63853aaca407c26f"
server_key_iv: "871720b753f451f86b03456a5a5b68b2"
server_key_version: "encryptionkey@0"
I20260927 18:43:31.488962 21887 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260927 18:43:31.493201 21902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:31.494886 21887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:43:31.495064 21887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "4bbc17c503cc452ab19404c94888f829"
format_stamp: "Formatted at 2026-09-27 18:43:31 on dist-test-slave-9jll"
server_key: "ae00e70f15fc454b63853aaca407c26f"
server_key_iv: "871720b753f451f86b03456a5a5b68b2"
server_key_version: "encryptionkey@0"
I20260927 18:43:31.495280 21887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:31.519737 21887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:31.524960 21887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:31.525279 21887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:31.526628 21887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:31.528261 21887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:31.528345 21887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:31.528442 21887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:31.528491 21887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:31.564056 21887 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:38173
I20260927 18:43:31.564110 22014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:38173 every 8 connection(s)
I20260927 18:43:31.565608 21887 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:43:31.568032 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 21887
I20260927 18:43:31.568257 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:43:31.568768 2429 external_mini_cluster.cc:1428] Setting key 842acd253fd66f6149af10868e2de845
Sep 27 18:43:31 dist-test-slave-9jll krb5kdc[21781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534611, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.65@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:43:31.589633 22017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:31.568947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:48555 (local address 127.2.95.126:45557)
0927 18:43:31.569503 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:31.569508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:31.570852 (+ 1344us) server_negotiation.cc:407] Connection header received
0927 18:43:31.572966 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:31.572974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:31.573102 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:31.573254 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:31.575097 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:31.576165 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:31.577951 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:31.578586 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:31.578747 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:31.582838 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:31.582844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:31.582848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:31.582892 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:31.585898 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:31.586781 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:31.586785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:31.586788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:31.586939 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:31.587441 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:31.587445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:31.587447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:31.587803 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:31.587966 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:31.588870 (+ 904us) server_negotiation.cc:299] Negotiation successful
0927 18:43:31.589154 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":267,"threads_started":1}
I20260927 18:43:31.590934 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:31.569635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45557 (local address 127.2.95.65:48555)
0927 18:43:31.570608 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:31.570669 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:31.572627 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:31.573468 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:31.573477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:31.574201 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:31.574902 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:31.574923 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:31.576370 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:31.576375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:31.577806 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:31.577820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:31.578844 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:31.578851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:31.579075 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:31.579841 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:31.579867 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:31.582545 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:31.586063 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:31.586076 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:31.586099 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:31.586631 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:31.587061 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:31.587067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:31.587072 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:31.587312 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:31.587949 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:31.587959 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:31.588422 (+ 463us) client_negotiation.cc:769] Sending connection context
0927 18:43:31.588728 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 18:43:31.589401 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":355,"threads_started":1}
I20260927 18:43:31.593147 22015 heartbeater.cc:347] Connected to a master server at 127.2.95.126:45557
I20260927 18:43:31.593577 22015 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:31.594507 22015 heartbeater.cc:511] Master 127.2.95.126:45557 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.95.66@KRBTEST.COM; defaulting to no policy
I20260927 18:43:31.597342 21826 ts_manager.cc:194] Registered new tserver with Master: 4bbc17c503cc452ab19404c94888f829 (127.2.95.65:38173)
I20260927 18:43:31.599208 21826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.65@KRBTEST.COM'} at 127.2.95.65:48555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
Entry for principal kudu/127.2.95.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab.
I20260927 18:43:31.634807 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:45557
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:32.021852 22021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:32.022219 22021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:32.022372 22021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:32.030473 22021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:32.030658 22021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:32.030826 22021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:43:32.042012 22021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.66.keytab
--principal=kudu/127.2.95.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:45557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:32.043879 22021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:32.045534 22021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:32.060562 22027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:32.060496 22029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:32.060526 22026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:32.061545 22021 server_base.cc:1034] running on GCE node
I20260927 18:43:32.062361 22021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:32.063575 22021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:32.064848 22021 hybrid_clock.cc:648] HybridClock initialized: now 1790534612064777 us; error 63 us; skew 500 ppm
I20260927 18:43:32.065213 22021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:32 dist-test-slave-9jll krb5kdc[21781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534612, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:32.068966 22021 init.cc:377] Logged in from keytab as kudu/127.2.95.66@KRBTEST.COM (short username kudu)
I20260927 18:43:32.070941 22021 webserver.cc:460] Webserver started at http://127.2.95.66:42263/ using document root <none> and password file <none>
I20260927 18:43:32.071811 22021 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:32.071928 22021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:32.072273 22021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:32.075933 22021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "08ae679e1c4a42699181ff2e29965c54"
format_stamp: "Formatted at 2026-09-27 18:43:32 on dist-test-slave-9jll"
server_key: "414f774255a64d9516b0b33dbe4a9b27"
server_key_iv: "42e84222a0ebf4efdfecc8e283bfb795"
server_key_version: "encryptionkey@0"
I20260927 18:43:32.076851 22021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "08ae679e1c4a42699181ff2e29965c54"
format_stamp: "Formatted at 2026-09-27 18:43:32 on dist-test-slave-9jll"
server_key: "414f774255a64d9516b0b33dbe4a9b27"
server_key_iv: "42e84222a0ebf4efdfecc8e283bfb795"
server_key_version: "encryptionkey@0"
I20260927 18:43:32.083243 22021 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 18:43:32.087631 22036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:32.089239 22021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:43:32.089433 22021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "08ae679e1c4a42699181ff2e29965c54"
format_stamp: "Formatted at 2026-09-27 18:43:32 on dist-test-slave-9jll"
server_key: "414f774255a64d9516b0b33dbe4a9b27"
server_key_iv: "42e84222a0ebf4efdfecc8e283bfb795"
server_key_version: "encryptionkey@0"
I20260927 18:43:32.089661 22021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:32.110476 22021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:32.115480 22021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:32.115811 22021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:32.116986 22021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:32.118809 22021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:32.118888 22021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:32.119019 22021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:32.119068 22021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:32.155339 22021 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:34609
I20260927 18:43:32.155380 22148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:34609 every 8 connection(s)
I20260927 18:43:32.156924 22021 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:43:32.166756 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 22021
I20260927 18:43:32.166985 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:43:32.167627 2429 external_mini_cluster.cc:1428] Setting key 6b655d687f8c67bf3c9a99179460b10d
Sep 27 18:43:32 dist-test-slave-9jll krb5kdc[21781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534612, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.66@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:43:32.181901 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:32.160112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:54791 (local address 127.2.95.126:45557)
0927 18:43:32.160645 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:32.160653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:32.161887 (+ 1234us) server_negotiation.cc:407] Connection header received
0927 18:43:32.164059 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:32.164067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:32.164185 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:32.164310 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:32.166132 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.167400 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.169256 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.169899 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.170042 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:32.174950 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:32.174957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:32.174961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:32.175004 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:32.177894 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:32.178885 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:32.178891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:32.178893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:32.179050 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:32.179601 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:32.179605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:32.179607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:32.179978 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:32.180131 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:32.181114 (+ 983us) server_negotiation.cc:299] Negotiation successful
0927 18:43:32.181428 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":240,"threads_started":1}
I20260927 18:43:32.183023 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:32.160810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45557 (local address 127.2.95.66:54791)
0927 18:43:32.161653 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:32.161754 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:32.163698 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:32.164573 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:32.164583 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:32.165216 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:32.165935 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:32.165956 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.167571 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.167575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:32.169097 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:32.169110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.170169 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.170178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:32.170540 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:32.171443 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:32.171485 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:32.174630 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:32.178057 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:32.178067 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:32.178092 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:32.178720 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:32.179159 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:32.179164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:32.179167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:32.179452 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:32.180103 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:32.180113 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:32.180647 (+ 534us) client_negotiation.cc:769] Sending connection context
0927 18:43:32.180992 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 18:43:32.181578 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":320,"threads_started":1}
I20260927 18:43:32.185104 22149 heartbeater.cc:347] Connected to a master server at 127.2.95.126:45557
I20260927 18:43:32.185480 22149 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:32.186440 22149 heartbeater.cc:511] Master 127.2.95.126:45557 requested a full tablet report, sending...
I20260927 18:43:32.188551 21826 ts_manager.cc:194] Registered new tserver with Master: 08ae679e1c4a42699181ff2e29965c54 (127.2.95.66:34609)
I20260927 18:43:32.189821 21826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.66@KRBTEST.COM'} at 127.2.95.66:54791
WARNING: no policy specified for kudu/127.2.95.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
Entry for principal kudu/127.2.95.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab.
I20260927 18:43:32.238142 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:45557
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:43:32.603034 22015 heartbeater.cc:503] Master 127.2.95.126:45557 was elected leader, sending a full tablet report...
W20260927 18:43:32.633235 22155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:32.633597 22155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:32.633744 22155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:32.641711 22155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:32.641867 22155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:32.642030 22155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:43:32.653786 22155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.67.keytab
--principal=kudu/127.2.95.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:45557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:32.655571 22155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:32.657253 22155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:32.672837 22163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:32.672891 22161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:32.672940 22160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:32.674077 22155 server_base.cc:1034] running on GCE node
I20260927 18:43:32.674840 22155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:32.675995 22155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:32.677241 22155 hybrid_clock.cc:648] HybridClock initialized: now 1790534612677152 us; error 61 us; skew 500 ppm
I20260927 18:43:32.677631 22155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:32 dist-test-slave-9jll krb5kdc[21781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534612, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:32.681346 22155 init.cc:377] Logged in from keytab as kudu/127.2.95.67@KRBTEST.COM (short username kudu)
I20260927 18:43:32.683054 22155 webserver.cc:460] Webserver started at http://127.2.95.67:41745/ using document root <none> and password file <none>
I20260927 18:43:32.683918 22155 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:32.684038 22155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:32.684363 22155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:32.687965 22155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "694c0595549a4d8ea0b3b4585eacadc8"
format_stamp: "Formatted at 2026-09-27 18:43:32 on dist-test-slave-9jll"
server_key: "b0c822dd9becc2e24bd609f82f124e74"
server_key_iv: "028b2fbd1a18e2841ce442af9054e859"
server_key_version: "encryptionkey@0"
I20260927 18:43:32.688881 22155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "694c0595549a4d8ea0b3b4585eacadc8"
format_stamp: "Formatted at 2026-09-27 18:43:32 on dist-test-slave-9jll"
server_key: "b0c822dd9becc2e24bd609f82f124e74"
server_key_iv: "028b2fbd1a18e2841ce442af9054e859"
server_key_version: "encryptionkey@0"
I20260927 18:43:32.695021 22155 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 18:43:32.699095 22170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:32.700688 22155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 18:43:32.700911 22155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "694c0595549a4d8ea0b3b4585eacadc8"
format_stamp: "Formatted at 2026-09-27 18:43:32 on dist-test-slave-9jll"
server_key: "b0c822dd9becc2e24bd609f82f124e74"
server_key_iv: "028b2fbd1a18e2841ce442af9054e859"
server_key_version: "encryptionkey@0"
I20260927 18:43:32.701122 22155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:32.722149 22155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:32.727228 22155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:32.727530 22155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:32.728693 22155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:32.730362 22155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:32.730458 22155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:32.730564 22155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:32.730608 22155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:32.762904 22155 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:41607
I20260927 18:43:32.762938 22282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:41607 every 8 connection(s)
I20260927 18:43:32.764432 22155 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:43:32.771008 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 22155
I20260927 18:43:32.771202 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:43:32.771807 2429 external_mini_cluster.cc:1428] Setting key 9ae208f7b1c6e8c861fc23d20538645e
Sep 27 18:43:32 dist-test-slave-9jll krb5kdc[21781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534612, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.67@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:43:32.788198 22285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:32.767548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:35319 (local address 127.2.95.126:45557)
0927 18:43:32.768181 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:32.768189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:32.769265 (+ 1076us) server_negotiation.cc:407] Connection header received
0927 18:43:32.771498 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:32.771505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:32.771617 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:32.771762 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:32.773985 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.775048 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.776839 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.777260 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.777387 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:32.781496 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:32.781502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:32.781506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:32.781546 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:32.784409 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:32.785248 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:32.785252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:32.785254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:32.785403 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:32.785963 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:32.785969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:32.785972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:32.786378 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:32.786550 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:32.787573 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0927 18:43:32.787818 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":320,"threads_started":1}
I20260927 18:43:32.789604 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:32.768158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45557 (local address 127.2.95.67:35319)
0927 18:43:32.769061 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:32.769114 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:32.771124 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:32.772050 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:32.772060 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:32.772813 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:32.773802 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:32.773834 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.775241 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.775248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:32.776685 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:32.776711 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.777490 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.777495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:32.777821 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:32.778616 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:43:32.778649 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:32.781228 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:32.784566 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:32.784575 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:32.784593 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:32.785143 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:32.785503 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:43:32.785507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:43:32.785509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:43:32.785827 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:43:32.786520 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:32.786530 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:32.787111 (+ 581us) client_negotiation.cc:769] Sending connection context
0927 18:43:32.787447 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 18:43:32.788076 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":323,"threads_started":1}
I20260927 18:43:32.791185 22283 heartbeater.cc:347] Connected to a master server at 127.2.95.126:45557
I20260927 18:43:32.791517 22283 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:32.792361 22283 heartbeater.cc:511] Master 127.2.95.126:45557 requested a full tablet report, sending...
I20260927 18:43:32.794080 21826 ts_manager.cc:194] Registered new tserver with Master: 694c0595549a4d8ea0b3b4585eacadc8 (127.2.95.67:41607)
I20260927 18:43:32.795096 21826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.95.67@KRBTEST.COM'} at 127.2.95.67:35319
I20260927 18:43:32.801133 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:43:32.820856 22285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:32.806064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.2.95.126:45557)
0927 18:43:32.806310 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:32.806319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:32.806679 (+ 360us) server_negotiation.cc:407] Connection header received
0927 18:43:32.806960 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:32.806967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:32.807084 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:32.807191 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:32.808838 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.809787 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.811237 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.811745 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.811870 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:32.814692 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:32.814697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:32.814700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:32.814741 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:32.817824 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:32.818602 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:32.818609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:32.818612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:32.818760 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:32.819153 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:32.819159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:32.819162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:32.819569 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:32.819856 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:32.820202 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 18:43:32.820485 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:43:32.839669 21826 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:43:32.848385 21826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:43:32.876050 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:32.863704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:38173 (local address 127.0.0.1:34312)
0927 18:43:32.864540 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:32.864593 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:32.864903 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:32.866066 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:32.866075 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:32.866119 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:32.866586 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:32.866599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.868817 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.868824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:32.871868 (+ 3044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:32.871894 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.874718 (+ 2824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.874725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:32.875024 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:32.875086 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 18:43:32.875264 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 18:43:32.875538 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":346,"threads_started":1}
I20260927 18:43:32.877444 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:32.864164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34312 (local address 127.2.95.65:38173)
0927 18:43:32.864896 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:32.864905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:32.864940 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:43:32.865152 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:32.865161 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:32.865452 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:32.865702 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:32.866792 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.868617 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.872157 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.874067 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.874217 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:32.874539 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:32.876772 (+ 2233us) server_negotiation.cc:299] Negotiation successful
0927 18:43:32.877035 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":369,"threads_started":1}
I20260927 18:43:32.884831 21950 tablet_service.cc:1467] Processing CreateTablet for tablet d8d41da5a63c4e3482b377f55ca36667 (DEFAULT_TABLE table=test-table [id=5874ba401fe74100a454d637646918b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:32.886353 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:32.865011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:34609 (local address 127.0.0.1:37866)
0927 18:43:32.866208 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:32.866226 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:32.866370 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:32.879133 (+ 12763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:32.879140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:32.879171 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:32.879649 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:32.879663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.881942 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.881948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:32.883578 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:32.883591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.885413 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.885420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:32.885603 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:32.885644 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 18:43:32.885736 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 18:43:32.885893 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":22,"thread_start_us":1424,"threads_started":1}
I20260927 18:43:32.886500 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:32.866727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:41607 (local address 127.0.0.1:39332)
0927 18:43:32.876392 (+ 9665us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:32.876417 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:32.876544 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:32.879528 (+ 2984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:32.879532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:32.879569 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:32.879883 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:32.879894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.881917 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.881923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:32.883922 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:32.883937 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.885769 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.885774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:32.885954 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:32.885983 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 18:43:32.886061 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 18:43:32.886195 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9537,"mutex_wait_us":25,"thread_start_us":12203,"threads_started":1}
I20260927 18:43:32.887002 21950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d41da5a63c4e3482b377f55ca36667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:32.887463 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:32.867218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39332 (local address 127.2.95.67:41607)
0927 18:43:32.878791 (+ 11573us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:32.878799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:32.878833 (+ 34us) server_negotiation.cc:407] Connection header received
0927 18:43:32.878935 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:32.878943 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:32.879158 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:32.879394 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:32.880059 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.881757 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.884072 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.885492 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.885606 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:32.886843 (+ 1237us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:32.886922 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 18:43:32.887093 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11384,"thread_start_us":11243,"threads_started":1}
I20260927 18:43:32.887470 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:32.865327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37866 (local address 127.2.95.66:34609)
0927 18:43:32.878274 (+ 12947us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:32.878286 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:32.878328 (+ 42us) server_negotiation.cc:407] Connection header received
0927 18:43:32.878472 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:32.878483 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:32.878856 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:32.879757 (+ 901us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:32.879988 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.881757 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.883739 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:32.885164 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:32.886663 (+ 1499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:32.886848 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:32.886922 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 18:43:32.887106 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12781,"thread_start_us":12636,"threads_started":1}
I20260927 18:43:32.891571 22218 tablet_service.cc:1467] Processing CreateTablet for tablet d8d41da5a63c4e3482b377f55ca36667 (DEFAULT_TABLE table=test-table [id=5874ba401fe74100a454d637646918b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:32.891609 22084 tablet_service.cc:1467] Processing CreateTablet for tablet d8d41da5a63c4e3482b377f55ca36667 (DEFAULT_TABLE table=test-table [id=5874ba401fe74100a454d637646918b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:32.893064 22218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d41da5a63c4e3482b377f55ca36667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:32.893215 22084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d41da5a63c4e3482b377f55ca36667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:32.904276 22302 tablet_bootstrap.cc:492] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829: Bootstrap starting.
I20260927 18:43:32.907050 22303 tablet_bootstrap.cc:492] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8: Bootstrap starting.
I20260927 18:43:32.908543 22302 tablet_bootstrap.cc:654] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:32.909821 22302 log.cc:824] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:32.910984 22303 tablet_bootstrap.cc:654] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:32.911803 22304 tablet_bootstrap.cc:492] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54: Bootstrap starting.
I20260927 18:43:32.912328 22303 log.cc:824] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:32.914273 22302 tablet_bootstrap.cc:492] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829: No bootstrap required, opened a new log
I20260927 18:43:32.914611 22302 ts_tablet_manager.cc:1397] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 18:43:32.915598 22304 tablet_bootstrap.cc:654] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:32.915766 22303 tablet_bootstrap.cc:492] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8: No bootstrap required, opened a new log
I20260927 18:43:32.916062 22303 ts_tablet_manager.cc:1397] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 18:43:32.916920 22304 log.cc:824] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:32.920217 22304 tablet_bootstrap.cc:492] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54: No bootstrap required, opened a new log
I20260927 18:43:32.920512 22304 ts_tablet_manager.cc:1397] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 18:43:32.923015 22302 raft_consensus.cc:348] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbc17c503cc452ab19404c94888f829" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 38173 } } peers { permanent_uuid: "08ae679e1c4a42699181ff2e29965c54" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 34609 } } peers { permanent_uuid: "694c0595549a4d8ea0b3b4585eacadc8" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41607 } }
I20260927 18:43:32.923447 22302 raft_consensus.cc:374] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:32.923552 22302 raft_consensus.cc:729] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbc17c503cc452ab19404c94888f829, State: Initialized, Role: FOLLOWER
I20260927 18:43:32.924216 22302 consensus_queue.cc:260] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbc17c503cc452ab19404c94888f829" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 38173 } } peers { permanent_uuid: "08ae679e1c4a42699181ff2e29965c54" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 34609 } } peers { permanent_uuid: "694c0595549a4d8ea0b3b4585eacadc8" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41607 } }
I20260927 18:43:32.926505 22302 ts_tablet_manager.cc:1428] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260927 18:43:32.929006 22303 raft_consensus.cc:348] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbc17c503cc452ab19404c94888f829" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 38173 } } peers { permanent_uuid: "08ae679e1c4a42699181ff2e29965c54" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 34609 } } peers { permanent_uuid: "694c0595549a4d8ea0b3b4585eacadc8" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41607 } }
I20260927 18:43:32.929551 22303 raft_consensus.cc:374] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:32.929646 22303 raft_consensus.cc:729] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694c0595549a4d8ea0b3b4585eacadc8, State: Initialized, Role: FOLLOWER
I20260927 18:43:32.930604 22303 consensus_queue.cc:260] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbc17c503cc452ab19404c94888f829" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 38173 } } peers { permanent_uuid: "08ae679e1c4a42699181ff2e29965c54" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 34609 } } peers { permanent_uuid: "694c0595549a4d8ea0b3b4585eacadc8" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41607 } }
I20260927 18:43:32.931169 22304 raft_consensus.cc:348] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbc17c503cc452ab19404c94888f829" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 38173 } } peers { permanent_uuid: "08ae679e1c4a42699181ff2e29965c54" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 34609 } } peers { permanent_uuid: "694c0595549a4d8ea0b3b4585eacadc8" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41607 } }
I20260927 18:43:32.931700 22304 raft_consensus.cc:374] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:32.931802 22304 raft_consensus.cc:729] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ae679e1c4a42699181ff2e29965c54, State: Initialized, Role: FOLLOWER
I20260927 18:43:32.931996 22283 heartbeater.cc:503] Master 127.2.95.126:45557 was elected leader, sending a full tablet report...
I20260927 18:43:32.932540 22304 consensus_queue.cc:260] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbc17c503cc452ab19404c94888f829" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 38173 } } peers { permanent_uuid: "08ae679e1c4a42699181ff2e29965c54" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 34609 } } peers { permanent_uuid: "694c0595549a4d8ea0b3b4585eacadc8" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41607 } }
I20260927 18:43:32.932751 22303 ts_tablet_manager.cc:1428] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260927 18:43:32.933821 22149 heartbeater.cc:503] Master 127.2.95.126:45557 was elected leader, sending a full tablet report...
I20260927 18:43:32.935524 22304 ts_tablet_manager.cc:1428] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260927 18:43:33.016494 22284 tablet.cc:2369] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:43:33.036509 22308 raft_consensus.cc:482] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:43:33.036940 22308 raft_consensus.cc:504] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbc17c503cc452ab19404c94888f829" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 38173 } } peers { permanent_uuid: "08ae679e1c4a42699181ff2e29965c54" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 34609 } } peers { permanent_uuid: "694c0595549a4d8ea0b3b4585eacadc8" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41607 } }
I20260927 18:43:33.039227 22308 leader_election.cc:288] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08ae679e1c4a42699181ff2e29965c54 (127.2.95.66:34609), 694c0595549a4d8ea0b3b4585eacadc8 (127.2.95.67:41607)
I20260927 18:43:33.048254 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:33.040818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:60333 (local address 127.2.95.67:41607)
0927 18:43:33.041060 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:33.041066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:33.041394 (+ 328us) server_negotiation.cc:407] Connection header received
0927 18:43:33.041534 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:33.041540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:33.041624 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:33.041728 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:33.042413 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.043236 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.045543 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.047003 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.047133 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:33.047264 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:33.047787 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 18:43:33.047984 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 18:43:33.048254 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:33.039851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:48693 (local address 127.2.95.66:34609)
0927 18:43:33.040138 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:33.040145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:33.040572 (+ 427us) server_negotiation.cc:407] Connection header received
0927 18:43:33.040829 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:33.040836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:33.040934 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:33.041039 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:33.041764 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.042851 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.045506 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.046941 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.047120 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:33.047258 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:33.047768 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 18:43:33.047930 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 18:43:33.048379 22312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:33.040600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:41607 (local address 127.2.95.65:60333)
0927 18:43:33.041279 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:33.041294 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:33.041407 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:33.041763 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:33.041768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:33.041863 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:33.042265 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:33.042278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.043365 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.043370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:33.045386 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:33.045398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.047288 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.047293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:33.047523 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:33.047600 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 18:43:33.047701 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 18:43:33.047857 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":27,"thread_start_us":345,"threads_started":1}
I20260927 18:43:33.048419 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:33.039736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:34609 (local address 127.2.95.65:48693)
0927 18:43:33.040435 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:33.040453 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:33.040643 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:33.041161 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:33.041165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:33.041205 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:33.041575 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:33.041588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.042973 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.042978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:33.045283 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:33.045300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.047274 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.047281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:33.047524 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:33.047594 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 18:43:33.047701 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 18:43:33.047917 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":63,"thread_start_us":369,"threads_started":1}
I20260927 18:43:33.049597 22104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d41da5a63c4e3482b377f55ca36667" candidate_uuid: "4bbc17c503cc452ab19404c94888f829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ae679e1c4a42699181ff2e29965c54" is_pre_election: true
I20260927 18:43:33.049692 22238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d41da5a63c4e3482b377f55ca36667" candidate_uuid: "4bbc17c503cc452ab19404c94888f829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694c0595549a4d8ea0b3b4585eacadc8" is_pre_election: true
I20260927 18:43:33.050232 22238 raft_consensus.cc:2454] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbc17c503cc452ab19404c94888f829 in term 0.
I20260927 18:43:33.050302 22104 raft_consensus.cc:2454] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbc17c503cc452ab19404c94888f829 in term 0.
I20260927 18:43:33.050973 21903 leader_election.cc:302] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbc17c503cc452ab19404c94888f829, 694c0595549a4d8ea0b3b4585eacadc8; no voters:
I20260927 18:43:33.051437 22308 raft_consensus.cc:2781] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:43:33.051586 22308 raft_consensus.cc:482] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:43:33.051694 22308 raft_consensus.cc:3037] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:33.054361 22308 raft_consensus.cc:504] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbc17c503cc452ab19404c94888f829" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 38173 } } peers { permanent_uuid: "08ae679e1c4a42699181ff2e29965c54" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 34609 } } peers { permanent_uuid: "694c0595549a4d8ea0b3b4585eacadc8" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41607 } }
I20260927 18:43:33.055048 22308 leader_election.cc:288] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [CANDIDATE]: Term 1 election: Requested vote from peers 08ae679e1c4a42699181ff2e29965c54 (127.2.95.66:34609), 694c0595549a4d8ea0b3b4585eacadc8 (127.2.95.67:41607)
I20260927 18:43:33.055627 22104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d41da5a63c4e3482b377f55ca36667" candidate_uuid: "4bbc17c503cc452ab19404c94888f829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ae679e1c4a42699181ff2e29965c54"
I20260927 18:43:33.055778 22238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d41da5a63c4e3482b377f55ca36667" candidate_uuid: "4bbc17c503cc452ab19404c94888f829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694c0595549a4d8ea0b3b4585eacadc8"
I20260927 18:43:33.055907 22104 raft_consensus.cc:3037] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:33.056025 22238 raft_consensus.cc:3037] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:33.058714 22104 raft_consensus.cc:2454] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbc17c503cc452ab19404c94888f829 in term 1.
I20260927 18:43:33.058764 22238 raft_consensus.cc:2454] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbc17c503cc452ab19404c94888f829 in term 1.
I20260927 18:43:33.059206 21903 leader_election.cc:302] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ae679e1c4a42699181ff2e29965c54, 4bbc17c503cc452ab19404c94888f829; no voters:
I20260927 18:43:33.059608 22308 raft_consensus.cc:2781] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:33.060238 22308 raft_consensus.cc:686] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 1 LEADER]: Becoming Leader. State: Replica: 4bbc17c503cc452ab19404c94888f829, State: Running, Role: LEADER
I20260927 18:43:33.060849 22308 consensus_queue.cc:237] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbc17c503cc452ab19404c94888f829" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 38173 } } peers { permanent_uuid: "08ae679e1c4a42699181ff2e29965c54" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 34609 } } peers { permanent_uuid: "694c0595549a4d8ea0b3b4585eacadc8" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41607 } }
I20260927 18:43:33.067240 21826 catalog_manager.cc:5495] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbc17c503cc452ab19404c94888f829 (127.2.95.65). New cstate: current_term: 1 leader_uuid: "4bbc17c503cc452ab19404c94888f829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bbc17c503cc452ab19404c94888f829" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08ae679e1c4a42699181ff2e29965c54" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 34609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "694c0595549a4d8ea0b3b4585eacadc8" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41607 } health_report { overall_health: UNKNOWN } } }
W20260927 18:43:33.068607 22016 tablet.cc:2369] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:43:33.122225 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:33.115892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.2.95.65:38173)
0927 18:43:33.116153 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:33.116160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:33.116241 (+ 81us) server_negotiation.cc:407] Connection header received
0927 18:43:33.116527 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:33.116531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:33.116630 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:33.116780 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:43:33.117227 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.118146 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.119621 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.120060 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.120177 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:33.120781 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:43:33.121519 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:43:33.121708 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:33.121810 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 18:43:33.121919 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 18:43:33.133226 22238 raft_consensus.cc:1261] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8 [term 1 FOLLOWER]: Refusing update from remote peer 4bbc17c503cc452ab19404c94888f829: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:33.133208 22104 raft_consensus.cc:1261] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54 [term 1 FOLLOWER]: Refusing update from remote peer 4bbc17c503cc452ab19404c94888f829: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:33.134369 22308 consensus_queue.cc:1035] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ae679e1c4a42699181ff2e29965c54" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 34609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:33.134747 22313 consensus_queue.cc:1035] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "694c0595549a4d8ea0b3b4585eacadc8" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:33.145918 22316 mvcc.cc:205] Tried to move back new op lower bound from 7334029775380766720 to 7334029775102287872. Current Snapshot: MvccSnapshot[applied={T|T < 7334029775380766720}]
I20260927 18:43:33.150672 22317 mvcc.cc:205] Tried to move back new op lower bound from 7334029775380766720 to 7334029775102287872. Current Snapshot: MvccSnapshot[applied={T|T < 7334029775380766720}]
I20260927 18:43:33.165091 21826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:33.165469 21826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:33.170550 21826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e09393565190402ebee12c3bfa12b178: Sending DeleteTablet for 3 replicas of tablet d8d41da5a63c4e3482b377f55ca36667
I20260927 18:43:33.172267 22218 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d41da5a63c4e3482b377f55ca36667 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:33 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:39332
I20260927 18:43:33.172768 21950 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d41da5a63c4e3482b377f55ca36667 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:33 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:34312
I20260927 18:43:33.173202 22084 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d41da5a63c4e3482b377f55ca36667 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:33 UTC) from {username='kudu', principal='kudu/127.2.95.126@KRBTEST.COM'} at 127.0.0.1:37866
I20260927 18:43:33.174347 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 21887
I20260927 18:43:33.176896 22326 tablet_replica.cc:330] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8: stopping tablet replica
I20260927 18:43:33.177529 22326 raft_consensus.cc:2229] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:33.178297 22326 raft_consensus.cc:2258] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:33.179582 22328 tablet_replica.cc:330] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54: stopping tablet replica
I20260927 18:43:33.180172 22328 raft_consensus.cc:2229] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:33.180830 22328 raft_consensus.cc:2258] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:33.181574 22326 ts_tablet_manager.cc:1905] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:33.182590 22327 tablet_replica.cc:330] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829: stopping tablet replica
I20260927 18:43:33.183110 22327 raft_consensus.cc:2229] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:43:33.184077 22327 raft_consensus.cc:2258] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:33.186715 22327 ts_tablet_manager.cc:1905] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:33.186710 22328 ts_tablet_manager.cc:1905] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:33.190145 22326 ts_tablet_manager.cc:1918] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:33.190464 22326 log.cc:1196] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/d8d41da5a63c4e3482b377f55ca36667
I20260927 18:43:33.191241 22326 ts_tablet_manager.cc:1939] T d8d41da5a63c4e3482b377f55ca36667 P 694c0595549a4d8ea0b3b4585eacadc8: Deleting consensus metadata
I20260927 18:43:33.193957 21810 catalog_manager.cc:4841] TS 694c0595549a4d8ea0b3b4585eacadc8 (127.2.95.67:41607): tablet d8d41da5a63c4e3482b377f55ca36667 (table test-table [id=5874ba401fe74100a454d637646918b9]) successfully deleted
I20260927 18:43:33.195773 22327 ts_tablet_manager.cc:1918] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:33.195983 22327 log.cc:1196] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/d8d41da5a63c4e3482b377f55ca36667
I20260927 18:43:33.196653 22327 ts_tablet_manager.cc:1939] T d8d41da5a63c4e3482b377f55ca36667 P 4bbc17c503cc452ab19404c94888f829: Deleting consensus metadata
I20260927 18:43:33.199110 21810 catalog_manager.cc:4841] TS 4bbc17c503cc452ab19404c94888f829 (127.2.95.65:38173): tablet d8d41da5a63c4e3482b377f55ca36667 (table test-table [id=5874ba401fe74100a454d637646918b9]) successfully deleted
I20260927 18:43:33.200402 22328 ts_tablet_manager.cc:1918] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:33.200611 22328 log.cc:1196] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/d8d41da5a63c4e3482b377f55ca36667
I20260927 18:43:33.201215 22328 ts_tablet_manager.cc:1939] T d8d41da5a63c4e3482b377f55ca36667 P 08ae679e1c4a42699181ff2e29965c54: Deleting consensus metadata
Sep 27 18:43:33 dist-test-slave-9jll krb5kdc[21781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.65@KRBTEST.COM
I20260927 18:43:33.203523 21810 catalog_manager.cc:4841] TS 08ae679e1c4a42699181ff2e29965c54 (127.2.95.66:34609): tablet d8d41da5a63c4e3482b377f55ca36667 (table test-table [id=5874ba401fe74100a454d637646918b9]) successfully deleted
I20260927 18:43:33.211771 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:33.175640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.2.95.65:38173)
0927 18:43:33.175872 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:33.175878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:33.181811 (+ 5933us) server_negotiation.cc:407] Connection header received
0927 18:43:33.182219 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:33.182225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:33.182351 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:33.182465 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:33.189943 (+ 7478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.191347 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.193645 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.194416 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.194557 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:33.203770 (+ 9213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:33.203791 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:33.203815 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:33.203870 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:33.207364 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:33.208501 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:33.208512 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:33.208514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:33.208740 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:33.209228 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:33.209232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:33.209234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:33.209796 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:33.210223 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:33.210704 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 18:43:33.211415 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:43:33.212589 22010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:33.324033 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 21887
I20260927 18:43:33.342927 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 22021
Sep 27 18:43:33 dist-test-slave-9jll krb5kdc[21781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.66@KRBTEST.COM
I20260927 18:43:33.360761 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:33.343904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.2.95.66:34609)
0927 18:43:33.344097 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:33.344102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:33.344125 (+ 23us) server_negotiation.cc:407] Connection header received
0927 18:43:33.344335 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:33.344340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:33.344418 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:33.344521 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:33.346058 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.346903 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.348287 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.348717 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.348844 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:33.353889 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:33.353910 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:33.353933 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:33.353981 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:33.357231 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:33.358290 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:33.358300 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:33.358303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:33.358482 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:33.359009 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:33.359015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:33.359018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:33.359561 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:33.359939 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:33.360229 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 18:43:33.360448 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:43:33.361542 22144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:33.462255 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 22021
I20260927 18:43:33.479269 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 22155
Sep 27 18:43:33 dist-test-slave-9jll krb5kdc[21781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.67@KRBTEST.COM
I20260927 18:43:33.497259 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:33.480224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.2.95.67:41607)
0927 18:43:33.480447 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:33.480453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:33.480478 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:43:33.480623 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:33.480627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:33.480796 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:33.480913 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:33.482451 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.483279 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.484638 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:33.485063 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:33.485179 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:33.490200 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:33.490222 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:33.490248 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:33.490295 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:33.493598 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:33.494664 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:33.494674 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:33.494677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:33.494854 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:33.495308 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:33.495312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:33.495315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:33.495807 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:33.496147 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:33.496536 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 18:43:33.496823 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 18:43:33.498122 22278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:33.594311 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 22155
I20260927 18:43:33.610402 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 21794
I20260927 18:43:33.611815 21856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:33.737949 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 21794
2026-09-27T18:43:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4831 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T18:43:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:33Z Disabled control of system clock
I20260927 18:43:33.827374 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:44807
--webserver_interface=127.2.95.126
--webserver_port=0
--builtin_ntp_servers=127.2.95.84:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:44807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:43:34.243733 22337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:34.244148 22337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:34.244279 22337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:34.252480 22337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:34.252637 22337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:34.252677 22337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:34.252734 22337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:34.252808 22337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:34.263234 22337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:44807
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:44807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:34.264987 22337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:34.267256 22337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:34.280988 22343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:34.281066 22342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:34.281023 22345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:34.281879 22337 server_base.cc:1034] running on GCE node
I20260927 18:43:34.282840 22337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:34.284682 22337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:34.285939 22337 hybrid_clock.cc:648] HybridClock initialized: now 1790534614285895 us; error 37 us; skew 500 ppm
I20260927 18:43:34.286351 22337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:43:34.289215 22337 webserver.cc:460] Webserver started at http://127.2.95.126:42525/ using document root <none> and password file <none>
I20260927 18:43:34.290108 22337 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:34.290206 22337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:34.290542 22337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:34.294054 22337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "8f723c53009b46ecb4ee186be528b123"
format_stamp: "Formatted at 2026-09-27 18:43:34 on dist-test-slave-9jll"
server_key: "55c0c1a934c830677d92edd3facd5b5d"
server_key_iv: "4afd32ab3a94091b616adf30f487a6e5"
server_key_version: "encryptionkey@0"
I20260927 18:43:34.294945 22337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "8f723c53009b46ecb4ee186be528b123"
format_stamp: "Formatted at 2026-09-27 18:43:34 on dist-test-slave-9jll"
server_key: "55c0c1a934c830677d92edd3facd5b5d"
server_key_iv: "4afd32ab3a94091b616adf30f487a6e5"
server_key_version: "encryptionkey@0"
I20260927 18:43:34.301036 22337 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:43:34.305176 22351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:34.307000 22337 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:43:34.307184 22337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "8f723c53009b46ecb4ee186be528b123"
format_stamp: "Formatted at 2026-09-27 18:43:34 on dist-test-slave-9jll"
server_key: "55c0c1a934c830677d92edd3facd5b5d"
server_key_iv: "4afd32ab3a94091b616adf30f487a6e5"
server_key_version: "encryptionkey@0"
I20260927 18:43:34.307402 22337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:34.343773 22337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:34.344887 22337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:34.345206 22337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:34.365996 22337 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:44807
I20260927 18:43:34.366020 22402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:44807 every 8 connection(s)
I20260927 18:43:34.367677 22337 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:34.371079 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 22337
I20260927 18:43:34.371264 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:34.371816 2429 external_mini_cluster.cc:1428] Setting key 7feaeb831ee21a4d57b8c7f9d0e77177
I20260927 18:43:34.372700 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:34.387369 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:34.374037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.2.95.126:44807)
0927 18:43:34.375192 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:34.375206 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:34.375323 (+ 117us) server_negotiation.cc:407] Connection header received
0927 18:43:34.377247 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:34.377275 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:34.377946 (+ 671us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:34.378399 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:34.379074 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:34.380572 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:34.382160 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:34.382994 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:34.383181 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:34.384593 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:34.384604 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:34.384623 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:34.385327 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:34.385431 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:34.385457 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:34.385574 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:34.385838 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 18:43:34.386218 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":374,"threads_started":1}
I20260927 18:43:34.388055 22403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123: Bootstrap starting.
I20260927 18:43:34.392350 22403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:34.393491 22403 log.cc:824] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:34.397128 22403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123: No bootstrap required, opened a new log
I20260927 18:43:34.405797 22403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f723c53009b46ecb4ee186be528b123" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 44807 } }
I20260927 18:43:34.406306 22403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:34.406427 22403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f723c53009b46ecb4ee186be528b123, State: Initialized, Role: FOLLOWER
I20260927 18:43:34.407145 22403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f723c53009b46ecb4ee186be528b123" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 44807 } }
I20260927 18:43:34.407431 22403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:34.407586 22403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:34.407768 22403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:34.410462 22403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f723c53009b46ecb4ee186be528b123" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 44807 } }
I20260927 18:43:34.411095 22403 leader_election.cc:302] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f723c53009b46ecb4ee186be528b123; no voters:
I20260927 18:43:34.411952 22403 leader_election.cc:288] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:34.412086 22408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:34.412817 22408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [term 1 LEADER]: Becoming Leader. State: Replica: 8f723c53009b46ecb4ee186be528b123, State: Running, Role: LEADER
I20260927 18:43:34.413451 22408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f723c53009b46ecb4ee186be528b123" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 44807 } }
I20260927 18:43:34.415400 22403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:34.416450 22410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f723c53009b46ecb4ee186be528b123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f723c53009b46ecb4ee186be528b123" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 44807 } } }
I20260927 18:43:34.417068 22410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:34.416797 22409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f723c53009b46ecb4ee186be528b123. Latest consensus state: current_term: 1 leader_uuid: "8f723c53009b46ecb4ee186be528b123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f723c53009b46ecb4ee186be528b123" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 44807 } } }
I20260927 18:43:34.419565 22409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:34.428575 22414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:34.435545 22414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:34.446957 22414 catalog_manager.cc:1347] Generated new cluster ID: b7296e608d744910b129291dce88d178
I20260927 18:43:34.447111 22414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:34.460047 22414 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:34.461828 22414 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:34.471961 22414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123: Generated new TSK 0
I20260927 18:43:34.473031 22414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:34.478142 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.65:0
--local_ip_for_outbound_sockets=127.2.95.65
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:44807
--builtin_ntp_servers=127.2.95.84:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 18:43:34.875870 22427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:34.876389 22427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:34.876560 22427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:34.885268 22427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:34.885442 22427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:34.885617 22427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.65
I20260927 18:43:34.896791 22427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.95.65
--webserver_port=0
--tserver_master_addrs=127.2.95.126:44807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.65
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:34.898562 22427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:34.900772 22427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:34.916178 22432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:34.916101 22435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:34.916163 22433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:34.917773 22427 server_base.cc:1034] running on GCE node
I20260927 18:43:34.918515 22427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:34.919802 22427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:34.921033 22427 hybrid_clock.cc:648] HybridClock initialized: now 1790534614920973 us; error 58 us; skew 500 ppm
I20260927 18:43:34.921427 22427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:43:34.924429 22427 webserver.cc:460] Webserver started at http://127.2.95.65:36021/ using document root <none> and password file <none>
I20260927 18:43:34.925284 22427 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:34.925400 22427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:34.925827 22427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:34.929344 22427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/data/instance:
uuid: "9d988ca9cbbe403f9ba28dae9e410c4b"
format_stamp: "Formatted at 2026-09-27 18:43:34 on dist-test-slave-9jll"
server_key: "71dff94e4d4d2727cd15a0d772e4fa86"
server_key_iv: "46ae3c7d9f44275322281f890cf7219e"
server_key_version: "encryptionkey@0"
I20260927 18:43:34.930403 22427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance:
uuid: "9d988ca9cbbe403f9ba28dae9e410c4b"
format_stamp: "Formatted at 2026-09-27 18:43:34 on dist-test-slave-9jll"
server_key: "71dff94e4d4d2727cd15a0d772e4fa86"
server_key_iv: "46ae3c7d9f44275322281f890cf7219e"
server_key_version: "encryptionkey@0"
I20260927 18:43:34.937705 22427 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260927 18:43:34.942628 22441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:34.944520 22427 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:43:34.944701 22427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/wal
uuid: "9d988ca9cbbe403f9ba28dae9e410c4b"
format_stamp: "Formatted at 2026-09-27 18:43:34 on dist-test-slave-9jll"
server_key: "71dff94e4d4d2727cd15a0d772e4fa86"
server_key_iv: "46ae3c7d9f44275322281f890cf7219e"
server_key_version: "encryptionkey@0"
I20260927 18:43:34.944936 22427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:34.963575 22427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:34.964604 22427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:34.964886 22427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:34.966019 22427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:34.967801 22427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:34.967902 22427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:34.967993 22427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:34.968050 22427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:35.003281 22427 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.65:40015
I20260927 18:43:35.003376 22553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.65:40015 every 8 connection(s)
I20260927 18:43:35.004928 22427 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/data/info.pb
I20260927 18:43:35.010128 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 22427
I20260927 18:43:35.010344 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/wal/instance
I20260927 18:43:35.010985 2429 external_mini_cluster.cc:1428] Setting key 5bf5d36467670d0de73f8afd58ced0ac
I20260927 18:43:35.014415 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.66:0
--local_ip_for_outbound_sockets=127.2.95.66
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:44807
--builtin_ntp_servers=127.2.95.84:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:43:35.020218 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:35.008271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:60621 (local address 127.2.95.126:44807)
0927 18:43:35.008878 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:35.008883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:35.010671 (+ 1788us) server_negotiation.cc:407] Connection header received
0927 18:43:35.012974 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:35.012980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:35.013074 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:35.013177 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:35.014518 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:35.015340 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:35.017059 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:35.017500 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:35.017617 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:35.019231 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:35.019236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:35.019239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:35.019261 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:35.019343 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:35.019358 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:35.019439 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:35.019687 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 18:43:35.019819 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":298,"threads_started":1}
I20260927 18:43:35.021447 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:35.009034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:44807 (local address 127.2.95.65:60621)
0927 18:43:35.010466 (+ 1432us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:35.010525 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:35.012682 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:35.013474 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:35.013485 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:35.013595 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:35.014348 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:35.014366 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:35.015493 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:35.015498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:35.016918 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:35.016936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:35.017758 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:35.017766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:35.018022 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:35.018867 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:35.018869 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:35.018925 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:35.019024 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:35.019452 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:35.019464 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:35.019473 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:43:35.019642 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 18:43:35.020153 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":446,"threads_started":1}
I20260927 18:43:35.023407 22554 heartbeater.cc:347] Connected to a master server at 127.2.95.126:44807
I20260927 18:43:35.023756 22554 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:35.024688 22554 heartbeater.cc:511] Master 127.2.95.126:44807 requested a full tablet report, sending...
I20260927 18:43:35.028052 22368 ts_manager.cc:194] Registered new tserver with Master: 9d988ca9cbbe403f9ba28dae9e410c4b (127.2.95.65:40015)
I20260927 18:43:35.030462 22368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.95.65:60621
W20260927 18:43:35.412506 22558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:35.412897 22558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:35.413007 22558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:35.421162 22558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:35.421305 22558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:35.421439 22558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.66
I20260927 18:43:35.431648 22558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.95.66
--webserver_port=0
--tserver_master_addrs=127.2.95.126:44807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.66
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:35.433451 22558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:35.435876 22558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:35.451898 22563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:35.451877 22564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:35.452077 22566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:35.452697 22558 server_base.cc:1034] running on GCE node
I20260927 18:43:35.453420 22558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:35.454646 22558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:35.455889 22558 hybrid_clock.cc:648] HybridClock initialized: now 1790534615455812 us; error 71 us; skew 500 ppm
I20260927 18:43:35.456255 22558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:43:35.458967 22558 webserver.cc:460] Webserver started at http://127.2.95.66:33239/ using document root <none> and password file <none>
I20260927 18:43:35.459795 22558 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:35.459916 22558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:35.460268 22558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:35.463651 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/data/instance:
uuid: "627ea6ee076a45af888ca1e795e69feb"
format_stamp: "Formatted at 2026-09-27 18:43:35 on dist-test-slave-9jll"
server_key: "983d0842517a57f048eb46f948a31545"
server_key_iv: "d3f2463bee16bdab6bc81919782abaac"
server_key_version: "encryptionkey@0"
I20260927 18:43:35.464545 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance:
uuid: "627ea6ee076a45af888ca1e795e69feb"
format_stamp: "Formatted at 2026-09-27 18:43:35 on dist-test-slave-9jll"
server_key: "983d0842517a57f048eb46f948a31545"
server_key_iv: "d3f2463bee16bdab6bc81919782abaac"
server_key_version: "encryptionkey@0"
I20260927 18:43:35.470616 22558 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 18:43:35.474586 22572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:35.476145 22558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:43:35.476325 22558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/wal
uuid: "627ea6ee076a45af888ca1e795e69feb"
format_stamp: "Formatted at 2026-09-27 18:43:35 on dist-test-slave-9jll"
server_key: "983d0842517a57f048eb46f948a31545"
server_key_iv: "d3f2463bee16bdab6bc81919782abaac"
server_key_version: "encryptionkey@0"
I20260927 18:43:35.476553 22558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:35.491168 22558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:35.492348 22558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:35.492713 22558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:35.494169 22558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:35.496501 22558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:35.496620 22558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:35.496744 22558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:35.496824 22558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:35.531399 22558 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.66:37469
I20260927 18:43:35.531445 22684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.66:37469 every 8 connection(s)
I20260927 18:43:35.532929 22558 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/data/info.pb
I20260927 18:43:35.540863 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 22558
I20260927 18:43:35.541049 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/wal/instance
I20260927 18:43:35.541532 2429 external_mini_cluster.cc:1428] Setting key b21722687b507dda62c16cd362893f6f
I20260927 18:43:35.544664 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.95.67:0
--local_ip_for_outbound_sockets=127.2.95.67
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:44807
--builtin_ntp_servers=127.2.95.84:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 18:43:35.547178 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:35.536051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.66:43685 (local address 127.2.95.126:44807)
0927 18:43:35.536580 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:35.536585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:35.538026 (+ 1441us) server_negotiation.cc:407] Connection header received
0927 18:43:35.540044 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:35.540051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:35.540145 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:35.540251 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:35.541556 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:35.542457 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:35.544079 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:35.544524 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:35.544637 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:35.546267 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:35.546272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:35.546276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:35.546295 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:35.546341 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:35.546354 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:35.546435 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:35.546660 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 18:43:35.546780 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":258,"threads_started":1}
I20260927 18:43:35.548547 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:35.536712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:44807 (local address 127.2.95.66:43685)
0927 18:43:35.537808 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:35.537889 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:35.539728 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:35.540537 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:35.540546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:35.540650 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:35.541371 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:35.541391 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:35.542612 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:35.542616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:35.543938 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:35.543951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:35.544753 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:35.544760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:35.545065 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:35.545921 (+ 856us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:35.545924 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:35.545977 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:35.546071 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:35.546445 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:35.546457 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:35.546466 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 18:43:35.546643 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 18:43:35.547110 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":306,"threads_started":1}
I20260927 18:43:35.550346 22685 heartbeater.cc:347] Connected to a master server at 127.2.95.126:44807
I20260927 18:43:35.550706 22685 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:35.551626 22685 heartbeater.cc:511] Master 127.2.95.126:44807 requested a full tablet report, sending...
I20260927 18:43:35.553354 22368 ts_manager.cc:194] Registered new tserver with Master: 627ea6ee076a45af888ca1e795e69feb (127.2.95.66:37469)
I20260927 18:43:35.554530 22368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.95.66:43685
W20260927 18:43:35.930315 22689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:35.930727 22689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:35.930853 22689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:35.939167 22689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:35.939347 22689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:35.939554 22689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.95.67
I20260927 18:43:35.951082 22689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.95.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.95.67
--webserver_port=0
--tserver_master_addrs=127.2.95.126:44807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.95.67
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:35.952847 22689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:35.955091 22689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:35.970440 22694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:35.970386 22695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:35.970343 22697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:35.971146 22689 server_base.cc:1034] running on GCE node
I20260927 18:43:35.971849 22689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:35.972985 22689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:35.974236 22689 hybrid_clock.cc:648] HybridClock initialized: now 1790534615974171 us; error 46 us; skew 500 ppm
I20260927 18:43:35.974594 22689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 18:43:35.977270 22689 webserver.cc:460] Webserver started at http://127.2.95.67:34813/ using document root <none> and password file <none>
I20260927 18:43:35.978230 22689 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:35.978339 22689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:35.978711 22689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:35.982290 22689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/data/instance:
uuid: "5db2be9ef48f4f17a6739c4d32a8aa80"
format_stamp: "Formatted at 2026-09-27 18:43:35 on dist-test-slave-9jll"
server_key: "90fcf607271b6f487c4f19946ff453fd"
server_key_iv: "00e6bad417b5c187913b75b00560c92a"
server_key_version: "encryptionkey@0"
I20260927 18:43:35.983141 22689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance:
uuid: "5db2be9ef48f4f17a6739c4d32a8aa80"
format_stamp: "Formatted at 2026-09-27 18:43:35 on dist-test-slave-9jll"
server_key: "90fcf607271b6f487c4f19946ff453fd"
server_key_iv: "00e6bad417b5c187913b75b00560c92a"
server_key_version: "encryptionkey@0"
I20260927 18:43:35.989125 22689 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:43:35.993152 22703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:35.994858 22689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:43:35.995031 22689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/wal
uuid: "5db2be9ef48f4f17a6739c4d32a8aa80"
format_stamp: "Formatted at 2026-09-27 18:43:35 on dist-test-slave-9jll"
server_key: "90fcf607271b6f487c4f19946ff453fd"
server_key_iv: "00e6bad417b5c187913b75b00560c92a"
server_key_version: "encryptionkey@0"
I20260927 18:43:35.995270 22689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:36.027144 22689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:36.028228 22689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:36.028532 22689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:36.029778 22689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 18:43:36.031565 22689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 18:43:36.031674 22689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:36.031803 22689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 18:43:36.031881 22689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:36.034847 22554 heartbeater.cc:503] Master 127.2.95.126:44807 was elected leader, sending a full tablet report...
I20260927 18:43:36.066609 22689 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.67:41525
I20260927 18:43:36.066663 22815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.67:41525 every 8 connection(s)
I20260927 18:43:36.068137 22689 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/data/info.pb
I20260927 18:43:36.078585 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 22689
I20260927 18:43:36.078780 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/wal/instance
I20260927 18:43:36.079282 2429 external_mini_cluster.cc:1428] Setting key bad6dc2d0d314562566533be45de79d7
I20260927 18:43:36.082574 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.071277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.67:55305 (local address 127.2.95.126:44807)
0927 18:43:36.071790 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:36.071795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:36.073274 (+ 1479us) server_negotiation.cc:407] Connection header received
0927 18:43:36.075317 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:36.075324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:36.075417 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:36.075525 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:36.076749 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.077563 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.079424 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.079871 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.079988 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.081499 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:36.081505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:36.081508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:36.081527 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:36.081573 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:36.081587 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:36.081662 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:36.081998 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 18:43:36.082115 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":256,"threads_started":1}
I20260927 18:43:36.083688 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.071795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:44807 (local address 127.2.95.67:55305)
0927 18:43:36.073072 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:36.073118 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:36.075005 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:36.075799 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:36.075808 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:36.075904 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:43:36.076550 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:36.076569 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.077745 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.077749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:36.079269 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:36.079284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.080104 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.080111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:36.080376 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.081140 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 18:43:36.081143 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:43:36.081196 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 18:43:36.081291 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:43:36.081785 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:43:36.081794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:43:36.081801 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 18:43:36.081971 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 18:43:36.082415 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":504,"threads_started":1}
I20260927 18:43:36.085280 22816 heartbeater.cc:347] Connected to a master server at 127.2.95.126:44807
I20260927 18:43:36.085650 22816 heartbeater.cc:464] Registering TS with master...
I20260927 18:43:36.086592 22816 heartbeater.cc:511] Master 127.2.95.126:44807 requested a full tablet report, sending...
I20260927 18:43:36.088274 22368 ts_manager.cc:194] Registered new tserver with Master: 5db2be9ef48f4f17a6739c4d32a8aa80 (127.2.95.67:41525)
I20260927 18:43:36.089226 22368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.95.67:55305
I20260927 18:43:36.095358 2429 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 18:43:36.106495 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.100103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.2.95.126:44807)
0927 18:43:36.100380 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:36.100395 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:36.100709 (+ 314us) server_negotiation.cc:407] Connection header received
0927 18:43:36.100901 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:36.100905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:36.100978 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:36.101053 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:36.101554 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.102524 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.103938 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.104370 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.104462 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.105633 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:36.105638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:36.105642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:36.105756 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:36.105797 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:36.105812 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:36.105892 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:36.106052 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 18:43:36.106142 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 18:43:36.111281 22368 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 18:43:36.119575 22368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 18:43:36.171525 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.134829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:41525 (local address 127.0.0.1:45666)
0927 18:43:36.160959 (+ 26130us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:36.160977 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:36.161102 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:36.163327 (+ 2225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:36.163339 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:36.163388 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:36.164525 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:36.164541 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.166601 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.166607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:36.168560 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:36.168573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.170647 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.170653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:36.170869 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.170942 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 18:43:36.171054 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 18:43:36.171233 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25989,"thread_start_us":25909,"threads_started":1}
I20260927 18:43:36.172742 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.135783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45666 (local address 127.2.95.67:41525)
0927 18:43:36.162458 (+ 26675us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:36.162470 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:36.162515 (+ 45us) server_negotiation.cc:407] Connection header received
0927 18:43:36.162632 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:36.162642 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:36.162969 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:36.163206 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:36.164727 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.166429 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.168785 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.170035 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.171965 (+ 1930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.172199 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:36.172274 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 18:43:36.172435 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26440,"thread_start_us":27659,"threads_started":1}
I20260927 18:43:36.173300 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.135892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:37469 (local address 127.0.0.1:60152)
0927 18:43:36.164059 (+ 28167us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:36.164076 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:36.164218 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:36.165155 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:36.165159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:36.165185 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:36.165631 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:36.165644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.167235 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.167245 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:36.169946 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:36.169962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.172346 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.172352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:36.172614 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.172669 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 18:43:36.172749 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 18:43:36.172910 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28016,"mutex_wait_us":196,"thread_start_us":23832,"threads_started":1}
I20260927 18:43:36.173447 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.134896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.2.95.65:40015)
0927 18:43:36.162478 (+ 27582us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:36.162488 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:36.162515 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:43:36.162633 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:36.162642 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:36.162968 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:36.163206 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:36.165070 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.167117 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.170226 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.171758 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.171883 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.172178 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:36.172876 (+ 698us) server_negotiation.cc:299] Negotiation successful
0927 18:43:36.173062 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27362,"thread_start_us":27346,"threads_started":1}
I20260927 18:43:36.173943 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.135535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.2.95.66:37469)
0927 18:43:36.161696 (+ 26161us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:36.161709 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:36.164214 (+ 2505us) server_negotiation.cc:407] Connection header received
0927 18:43:36.164380 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:36.164389 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:36.164640 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:36.164866 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:36.165817 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.167091 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.170214 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.171754 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.171867 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.172143 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:36.173084 (+ 941us) server_negotiation.cc:299] Negotiation successful
0927 18:43:36.173281 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25924,"thread_start_us":26168,"threads_started":1}
I20260927 18:43:36.174062 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.134515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.65:40015 (local address 127.0.0.1:52800)
0927 18:43:36.160179 (+ 25664us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:36.160238 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:36.160536 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:36.164388 (+ 3852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:36.164392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:36.164416 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:36.164842 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:36.164857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.167262 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.167269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:36.169947 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:36.169962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.172332 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.172339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:36.172636 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.172671 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 18:43:36.172750 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 18:43:36.173780 (+ 1030us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25246,"mutex_wait_us":1000,"thread_start_us":29247,"threads_started":1}
I20260927 18:43:36.177527 22751 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7b19615e604ce2bc37e5d1a9538534 (DEFAULT_TABLE table=test-table [id=8bb3df3a259a4f00b7a9335205cc7ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:36.177935 22489 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7b19615e604ce2bc37e5d1a9538534 (DEFAULT_TABLE table=test-table [id=8bb3df3a259a4f00b7a9335205cc7ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:36.177935 22620 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7b19615e604ce2bc37e5d1a9538534 (DEFAULT_TABLE table=test-table [id=8bb3df3a259a4f00b7a9335205cc7ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 18:43:36.179100 22751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7b19615e604ce2bc37e5d1a9538534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:36.179373 22620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7b19615e604ce2bc37e5d1a9538534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:36.179380 22489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7b19615e604ce2bc37e5d1a9538534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:36.192724 22835 tablet_bootstrap.cc:492] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80: Bootstrap starting.
I20260927 18:43:36.197538 22835 tablet_bootstrap.cc:654] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:36.198936 22835 log.cc:824] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:36.200822 22836 tablet_bootstrap.cc:492] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb: Bootstrap starting.
I20260927 18:43:36.203238 22835 tablet_bootstrap.cc:492] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80: No bootstrap required, opened a new log
I20260927 18:43:36.203703 22835 ts_tablet_manager.cc:1397] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 18:43:36.205200 22836 tablet_bootstrap.cc:654] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:36.206650 22837 tablet_bootstrap.cc:492] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b: Bootstrap starting.
I20260927 18:43:36.206750 22836 log.cc:824] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:36.211212 22837 tablet_bootstrap.cc:654] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:36.211486 22836 tablet_bootstrap.cc:492] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb: No bootstrap required, opened a new log
I20260927 18:43:36.211829 22836 ts_tablet_manager.cc:1397] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 18:43:36.212601 22837 log.cc:824] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:36.214741 22835 raft_consensus.cc:348] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 40015 } } peers { permanent_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41525 } } peers { permanent_uuid: "627ea6ee076a45af888ca1e795e69feb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37469 } }
I20260927 18:43:36.215348 22835 raft_consensus.cc:374] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:36.215483 22835 raft_consensus.cc:729] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db2be9ef48f4f17a6739c4d32a8aa80, State: Initialized, Role: FOLLOWER
I20260927 18:43:36.216418 22835 consensus_queue.cc:260] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 40015 } } peers { permanent_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41525 } } peers { permanent_uuid: "627ea6ee076a45af888ca1e795e69feb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37469 } }
I20260927 18:43:36.217254 22837 tablet_bootstrap.cc:492] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b: No bootstrap required, opened a new log
I20260927 18:43:36.217598 22837 ts_tablet_manager.cc:1397] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 18:43:36.217900 22816 heartbeater.cc:503] Master 127.2.95.126:44807 was elected leader, sending a full tablet report...
I20260927 18:43:36.218699 22835 ts_tablet_manager.cc:1428] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 18:43:36.221923 22836 raft_consensus.cc:348] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 40015 } } peers { permanent_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41525 } } peers { permanent_uuid: "627ea6ee076a45af888ca1e795e69feb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37469 } }
I20260927 18:43:36.222373 22836 raft_consensus.cc:374] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:36.222484 22836 raft_consensus.cc:729] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627ea6ee076a45af888ca1e795e69feb, State: Initialized, Role: FOLLOWER
I20260927 18:43:36.223263 22836 consensus_queue.cc:260] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 40015 } } peers { permanent_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41525 } } peers { permanent_uuid: "627ea6ee076a45af888ca1e795e69feb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37469 } }
I20260927 18:43:36.224645 22685 heartbeater.cc:503] Master 127.2.95.126:44807 was elected leader, sending a full tablet report...
I20260927 18:43:36.225548 22836 ts_tablet_manager.cc:1428] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 18:43:36.231899 22837 raft_consensus.cc:348] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 40015 } } peers { permanent_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41525 } } peers { permanent_uuid: "627ea6ee076a45af888ca1e795e69feb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37469 } }
I20260927 18:43:36.232422 22837 raft_consensus.cc:374] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:36.232517 22837 raft_consensus.cc:729] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d988ca9cbbe403f9ba28dae9e410c4b, State: Initialized, Role: FOLLOWER
I20260927 18:43:36.233240 22837 consensus_queue.cc:260] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 40015 } } peers { permanent_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41525 } } peers { permanent_uuid: "627ea6ee076a45af888ca1e795e69feb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37469 } }
I20260927 18:43:36.235170 22837 ts_tablet_manager.cc:1428] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b: Time spent starting tablet: real 0.017s user 0.008s sys 0.004s
W20260927 18:43:36.257864 22555 tablet.cc:2369] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:43:36.285456 22686 tablet.cc:2369] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 18:43:36.320247 22817 tablet.cc:2369] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:43:36.498274 22843 raft_consensus.cc:482] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 18:43:36.498745 22843 raft_consensus.cc:504] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 40015 } } peers { permanent_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41525 } } peers { permanent_uuid: "627ea6ee076a45af888ca1e795e69feb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37469 } }
I20260927 18:43:36.501003 22843 leader_election.cc:288] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5db2be9ef48f4f17a6739c4d32a8aa80 (127.2.95.67:41525), 627ea6ee076a45af888ca1e795e69feb (127.2.95.66:37469)
I20260927 18:43:36.509346 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.501766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:51115 (local address 127.2.95.66:37469)
0927 18:43:36.502047 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:36.502054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:36.503220 (+ 1166us) server_negotiation.cc:407] Connection header received
0927 18:43:36.503436 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:36.503442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:36.503560 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:36.503676 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:36.504463 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.505301 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.507423 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.508235 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.508362 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.508481 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:36.508881 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 18:43:36.509030 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 18:43:36.509338 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.501577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.95.65:60345 (local address 127.2.95.67:41525)
0927 18:43:36.501823 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:36.501830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:36.503000 (+ 1170us) server_negotiation.cc:407] Connection header received
0927 18:43:36.503224 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:36.503231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:36.503323 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:36.503425 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 18:43:36.504215 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.505064 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.507447 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.508239 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.508374 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.508490 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:36.508912 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 18:43:36.509072 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 18:43:36.509486 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.501613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.66:37469 (local address 127.2.95.65:51115)
0927 18:43:36.503058 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:36.503078 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:36.503261 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:36.503707 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:36.503710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:36.503735 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:36.504214 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:36.504227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.505456 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.505460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:36.507191 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:36.507207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.508464 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.508471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:36.508667 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.508724 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 18:43:36.508805 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 18:43:36.508975 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1308,"mutex_wait_us":337,"thread_start_us":648,"threads_started":1}
I20260927 18:43:36.510174 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.501606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.67:41525 (local address 127.2.95.65:60345)
0927 18:43:36.502859 (+ 1253us) negotiation.cc:107] Waiting for socket to connect
0927 18:43:36.502878 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 18:43:36.503051 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:43:36.503473 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:43:36.503478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:43:36.503508 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 18:43:36.504032 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:36.504045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.505220 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.505226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:36.507216 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:43:36.507231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.508521 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.508529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:43:36.508719 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.508750 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 18:43:36.508866 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 18:43:36.509031 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":71,"thread_start_us":676,"threads_started":1}
I20260927 18:43:36.510990 22640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7b19615e604ce2bc37e5d1a9538534" candidate_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "627ea6ee076a45af888ca1e795e69feb" is_pre_election: true
I20260927 18:43:36.511518 22771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7b19615e604ce2bc37e5d1a9538534" candidate_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80" is_pre_election: true
I20260927 18:43:36.511679 22640 raft_consensus.cc:2454] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d988ca9cbbe403f9ba28dae9e410c4b in term 0.
I20260927 18:43:36.512109 22771 raft_consensus.cc:2454] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d988ca9cbbe403f9ba28dae9e410c4b in term 0.
I20260927 18:43:36.512579 22442 leader_election.cc:302] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 627ea6ee076a45af888ca1e795e69feb, 9d988ca9cbbe403f9ba28dae9e410c4b; no voters:
I20260927 18:43:36.513084 22843 raft_consensus.cc:2781] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 18:43:36.513252 22843 raft_consensus.cc:482] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 18:43:36.513376 22843 raft_consensus.cc:3037] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:36.515951 22843 raft_consensus.cc:504] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 40015 } } peers { permanent_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41525 } } peers { permanent_uuid: "627ea6ee076a45af888ca1e795e69feb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37469 } }
I20260927 18:43:36.516566 22843 leader_election.cc:288] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [CANDIDATE]: Term 1 election: Requested vote from peers 5db2be9ef48f4f17a6739c4d32a8aa80 (127.2.95.67:41525), 627ea6ee076a45af888ca1e795e69feb (127.2.95.66:37469)
I20260927 18:43:36.517177 22771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7b19615e604ce2bc37e5d1a9538534" candidate_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80"
I20260927 18:43:36.517302 22640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7b19615e604ce2bc37e5d1a9538534" candidate_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "627ea6ee076a45af888ca1e795e69feb"
I20260927 18:43:36.517483 22771 raft_consensus.cc:3037] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:36.517580 22640 raft_consensus.cc:3037] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:36.521245 22640 raft_consensus.cc:2454] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d988ca9cbbe403f9ba28dae9e410c4b in term 1.
I20260927 18:43:36.521245 22771 raft_consensus.cc:2454] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d988ca9cbbe403f9ba28dae9e410c4b in term 1.
I20260927 18:43:36.521976 22444 leader_election.cc:302] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5db2be9ef48f4f17a6739c4d32a8aa80, 9d988ca9cbbe403f9ba28dae9e410c4b; no voters:
I20260927 18:43:36.522401 22843 raft_consensus.cc:2781] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:36.522994 22843 raft_consensus.cc:686] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 1 LEADER]: Becoming Leader. State: Replica: 9d988ca9cbbe403f9ba28dae9e410c4b, State: Running, Role: LEADER
I20260927 18:43:36.523584 22843 consensus_queue.cc:237] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 40015 } } peers { permanent_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41525 } } peers { permanent_uuid: "627ea6ee076a45af888ca1e795e69feb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37469 } }
I20260927 18:43:36.529033 22368 catalog_manager.cc:5495] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d988ca9cbbe403f9ba28dae9e410c4b (127.2.95.65). New cstate: current_term: 1 leader_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d988ca9cbbe403f9ba28dae9e410c4b" member_type: VOTER last_known_addr { host: "127.2.95.65" port: 40015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "627ea6ee076a45af888ca1e795e69feb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37469 } health_report { overall_health: UNKNOWN } } }
I20260927 18:43:36.550161 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.543894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52814 (local address 127.2.95.65:40015)
0927 18:43:36.544153 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:36.544161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:36.544185 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:43:36.544325 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:36.544330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:36.544484 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:36.544695 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:43:36.545140 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.545983 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.547490 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.547942 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.548044 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.548793 (+ 749us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:43:36.549517 (+ 724us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:43:36.549648 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:36.549817 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 18:43:36.549920 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 18:43:36.560595 22640 raft_consensus.cc:1261] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb [term 1 FOLLOWER]: Refusing update from remote peer 9d988ca9cbbe403f9ba28dae9e410c4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:36.560883 22771 raft_consensus.cc:1261] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80 [term 1 FOLLOWER]: Refusing update from remote peer 9d988ca9cbbe403f9ba28dae9e410c4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:43:36.561919 22846 consensus_queue.cc:1035] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "627ea6ee076a45af888ca1e795e69feb" member_type: VOTER last_known_addr { host: "127.2.95.66" port: 37469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:36.562387 22843 consensus_queue.cc:1035] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db2be9ef48f4f17a6739c4d32a8aa80" member_type: VOTER last_known_addr { host: "127.2.95.67" port: 41525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:43:36.576709 22849 mvcc.cc:205] Tried to move back new op lower bound from 7334029789419581440 to 7334029789285064704. Current Snapshot: MvccSnapshot[applied={T|T < 7334029789419581440}]
I20260927 18:43:36.594403 22367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:36.594722 22367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 18:43:36.599745 22367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f723c53009b46ecb4ee186be528b123: Sending DeleteTablet for 3 replicas of tablet 8c7b19615e604ce2bc37e5d1a9538534
I20260927 18:43:36.601316 22489 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7b19615e604ce2bc37e5d1a9538534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:36 UTC) from {username='slave'} at 127.0.0.1:52800
I20260927 18:43:36.601379 22751 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7b19615e604ce2bc37e5d1a9538534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:36 UTC) from {username='slave'} at 127.0.0.1:45666
I20260927 18:43:36.602165 22620 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7b19615e604ce2bc37e5d1a9538534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 18:43:36 UTC) from {username='slave'} at 127.0.0.1:60152
I20260927 18:43:36.602443 22859 tablet_replica.cc:330] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b: stopping tablet replica
I20260927 18:43:36.602995 22859 raft_consensus.cc:2229] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 1 LEADER]: Raft consensus shutting down.
I20260927 18:43:36.603888 22859 raft_consensus.cc:2258] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:36.605113 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 22427
I20260927 18:43:36.606854 22859 ts_tablet_manager.cc:1905] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:36.611677 22861 tablet_replica.cc:330] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb: stopping tablet replica
I20260927 18:43:36.612515 22861 raft_consensus.cc:2229] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:36.613281 22861 raft_consensus.cc:2258] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:36.617285 22860 tablet_replica.cc:330] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80: stopping tablet replica
I20260927 18:43:36.617892 22860 raft_consensus.cc:2229] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 18:43:36.618434 22859 ts_tablet_manager.cc:1918] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:36.618525 22860 raft_consensus.cc:2258] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 18:43:36.618667 22859 log.cc:1196] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-0/wal/wals/8c7b19615e604ce2bc37e5d1a9538534
I20260927 18:43:36.618757 22861 ts_tablet_manager.cc:1905] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:36.619441 22859 ts_tablet_manager.cc:1939] T 8c7b19615e604ce2bc37e5d1a9538534 P 9d988ca9cbbe403f9ba28dae9e410c4b: Deleting consensus metadata
I20260927 18:43:36.620791 22860 ts_tablet_manager.cc:1905] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 18:43:36.623055 22355 catalog_manager.cc:4841] TS 9d988ca9cbbe403f9ba28dae9e410c4b (127.2.95.65:40015): tablet 8c7b19615e604ce2bc37e5d1a9538534 (table test-table [id=8bb3df3a259a4f00b7a9335205cc7ddb]) successfully deleted
I20260927 18:43:36.626626 22861 ts_tablet_manager.cc:1918] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:36.626820 22861 log.cc:1196] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-1/wal/wals/8c7b19615e604ce2bc37e5d1a9538534
I20260927 18:43:36.627400 22861 ts_tablet_manager.cc:1939] T 8c7b19615e604ce2bc37e5d1a9538534 P 627ea6ee076a45af888ca1e795e69feb: Deleting consensus metadata
I20260927 18:43:36.628135 22860 ts_tablet_manager.cc:1918] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 18:43:36.628302 22860 log.cc:1196] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80: Deleting WAL directory at /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790534284715420-2429-0/minicluster-data/ts-2/wal/wals/8c7b19615e604ce2bc37e5d1a9538534
I20260927 18:43:36.628899 22860 ts_tablet_manager.cc:1939] T 8c7b19615e604ce2bc37e5d1a9538534 P 5db2be9ef48f4f17a6739c4d32a8aa80: Deleting consensus metadata
I20260927 18:43:36.629922 22352 catalog_manager.cc:4841] TS 627ea6ee076a45af888ca1e795e69feb (127.2.95.66:37469): tablet 8c7b19615e604ce2bc37e5d1a9538534 (table test-table [id=8bb3df3a259a4f00b7a9335205cc7ddb]) successfully deleted
I20260927 18:43:36.630447 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.606500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52816 (local address 127.2.95.65:40015)
0927 18:43:36.606739 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:36.606747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:36.618342 (+ 11595us) server_negotiation.cc:407] Connection header received
0927 18:43:36.618767 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:36.618773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:36.618894 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:36.619020 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:36.619921 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.621300 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.625527 (+ 4227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.626236 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.626344 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.628220 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:36.628233 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:36.628256 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:36.629188 (+ 932us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:36.629268 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:36.629295 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:36.629404 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:36.629899 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 18:43:36.630030 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 18:43:36.630893 22354 catalog_manager.cc:4841] TS 5db2be9ef48f4f17a6739c4d32a8aa80 (127.2.95.67:41525): tablet 8c7b19615e604ce2bc37e5d1a9538534 (table test-table [id=8bb3df3a259a4f00b7a9335205cc7ddb]) successfully deleted
I20260927 18:43:36.631721 22549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:36.742174 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 22427
I20260927 18:43:36.762028 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 22558
I20260927 18:43:36.769997 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.763072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.2.95.66:37469)
0927 18:43:36.763296 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:36.763301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:36.763323 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:43:36.763572 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:36.763578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:36.763696 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:36.763803 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:36.764292 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.765097 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.766682 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.767165 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.767284 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.768395 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:36.768406 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:36.768427 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:36.769084 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:36.769140 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:36.769162 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:36.769262 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:36.769443 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 18:43:36.769562 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 18:43:36.770838 22680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:36.863250 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 22558
I20260927 18:43:36.881201 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 22689
I20260927 18:43:36.888640 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:36.882244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.2.95.67:41525)
0927 18:43:36.882561 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:36.882567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:36.882592 (+ 25us) server_negotiation.cc:407] Connection header received
0927 18:43:36.882664 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:36.882669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:36.882768 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:36.882886 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:36.883346 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.884139 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.885490 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:36.885942 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:36.886061 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:36.887178 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:36.887186 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:36.887201 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:36.887860 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:36.887913 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:36.887934 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:36.888029 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:36.888128 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 18:43:36.888225 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260927 18:43:36.889488 22811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:36.982717 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 22689
I20260927 18:43:36.998898 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 22337
I20260927 18:43:37.000293 22398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:37.128983 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 22337
2026-09-27T18:43:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3388 ms)
[----------] 6 tests from EncryptionPolicyTest (24729 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:37 dist-test-slave-9jll krb5kdc[22869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:37 dist-test-slave-9jll krb5kdc[22869](info): set up 2 sockets
Sep 27 18:43:37 dist-test-slave-9jll krb5kdc[22869](info): commencing operation
krb5kdc: starting...
W20260927 18:43:38.186223 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:38 dist-test-slave-9jll krb5kdc[22869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:43:38.494026 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:38.882099 22882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:38.882462 22882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:38.882611 22882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:38.890439 22882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:38.890574 22882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:38.890650 22882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:38.890738 22882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:38.890801 22882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:38.901437 22882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:38.903188 22882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:38.904832 22882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:38.918318 22887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:38.918251 22888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:38.918298 22890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:38.918864 22882 server_base.cc:1034] running on GCE node
I20260927 18:43:38.919752 22882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:38.921422 22882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:38.922621 22882 hybrid_clock.cc:648] HybridClock initialized: now 1790534618922542 us; error 70 us; skew 500 ppm
I20260927 18:43:38.923075 22882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:38 dist-test-slave-9jll krb5kdc[22869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:38.932541 22882 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:43:38.934625 22882 webserver.cc:460] Webserver started at http://127.0.0.1:43933/ using document root <none> and password file <none>
I20260927 18:43:38.935529 22882 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:38.935645 22882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:38.935969 22882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:38.939990 22882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "6f8b5609015a4f46863b8a617a47c94b"
format_stamp: "Formatted at 2026-09-27 18:43:38 on dist-test-slave-9jll"
server_key: "83b77051b0fa8d421b560ea120fb0c56"
server_key_iv: "9f022990c365d39181d7ca08a194e868"
server_key_version: "encryptionkey@0"
I20260927 18:43:38.940944 22882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "6f8b5609015a4f46863b8a617a47c94b"
format_stamp: "Formatted at 2026-09-27 18:43:38 on dist-test-slave-9jll"
server_key: "83b77051b0fa8d421b560ea120fb0c56"
server_key_iv: "9f022990c365d39181d7ca08a194e868"
server_key_version: "encryptionkey@0"
I20260927 18:43:38.947536 22882 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260927 18:43:38.951861 22897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:38.953920 22882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:43:38.954139 22882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "6f8b5609015a4f46863b8a617a47c94b"
format_stamp: "Formatted at 2026-09-27 18:43:38 on dist-test-slave-9jll"
server_key: "83b77051b0fa8d421b560ea120fb0c56"
server_key_iv: "9f022990c365d39181d7ca08a194e868"
server_key_version: "encryptionkey@0"
I20260927 18:43:38.954448 22882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:38.981832 22882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:38.986871 22882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:38.987145 22882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:39.006927 22882 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45793
I20260927 18:43:39.006937 22948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45793 every 8 connection(s)
I20260927 18:43:39.008632 22882 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:39.013621 22949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:39.016124 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 22882
I20260927 18:43:39.016350 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:39.017089 2429 external_mini_cluster.cc:1428] Setting key a99d5a7b9ad0a768317c248b0ad1267c
I20260927 18:43:39.029474 22949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b: Bootstrap starting.
I20260927 18:43:39.033931 22949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b: Neither blocks nor log segments found. Creating new log.
Sep 27 18:43:39 dist-test-slave-9jll krb5kdc[22869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:43:39.035394 22949 log.cc:824] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:39.039690 22949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b: No bootstrap required, opened a new log
I20260927 18:43:39.045625 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:39.019733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57055 (local address 127.0.0.1:45793)
0927 18:43:39.020734 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:39.020758 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:39.020825 (+ 67us) server_negotiation.cc:407] Connection header received
0927 18:43:39.023474 (+ 2649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:39.023510 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:39.024140 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:39.024661 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:39.026393 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:39.028563 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:39.030155 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:39.031259 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:39.031482 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:39.036208 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:39.036224 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:39.036246 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:39.036296 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:39.040635 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:39.041557 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:39.041566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:39.041576 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:39.041846 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:39.042382 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:39.042388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:39.042391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:39.043138 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:39.043490 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:39.044057 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 18:43:39.044403 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":393,"threads_started":1}
I20260927 18:43:39.050130 22949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8b5609015a4f46863b8a617a47c94b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45793 } }
I20260927 18:43:39.050480 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:43:39.050633 22949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:39.050738 22949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8b5609015a4f46863b8a617a47c94b, State: Initialized, Role: FOLLOWER
I20260927 18:43:39.051421 22949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8b5609015a4f46863b8a617a47c94b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45793 } }
I20260927 18:43:39.051704 22949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:39.051851 22949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:39.052019 22949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:39.054615 22949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8b5609015a4f46863b8a617a47c94b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45793 } }
I20260927 18:43:39.055225 22949 leader_election.cc:302] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f8b5609015a4f46863b8a617a47c94b; no voters:
I20260927 18:43:39.055924 22949 leader_election.cc:288] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:39.056111 22954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:39.057262 22954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [term 1 LEADER]: Becoming Leader. State: Replica: 6f8b5609015a4f46863b8a617a47c94b, State: Running, Role: LEADER
I20260927 18:43:39.057989 22954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8b5609015a4f46863b8a617a47c94b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45793 } }
I20260927 18:43:39.058712 22949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:39.061398 22956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f8b5609015a4f46863b8a617a47c94b. Latest consensus state: current_term: 1 leader_uuid: "6f8b5609015a4f46863b8a617a47c94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8b5609015a4f46863b8a617a47c94b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45793 } } }
I20260927 18:43:39.061928 22956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:39.062132 22955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f8b5609015a4f46863b8a617a47c94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8b5609015a4f46863b8a617a47c94b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45793 } } }
I20260927 18:43:39.062481 22955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:39.076074 22966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:39.083740 22966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:39.097487 22966 catalog_manager.cc:1347] Generated new cluster ID: cd2ca6265ace4f7eb15a72454b29dcd2
I20260927 18:43:39.097630 22966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:39.105422 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:39.077558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39449 (local address 127.0.0.1:45793)
0927 18:43:39.077803 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:39.077811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:39.078765 (+ 954us) server_negotiation.cc:407] Connection header received
0927 18:43:39.079062 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:39.079070 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:39.079196 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:39.079344 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:39.082272 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:39.083781 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:39.086050 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:39.086868 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:39.086994 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:39.092219 (+ 5225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:39.092226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:39.092235 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:39.092288 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:39.096274 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:39.101946 (+ 5672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:39.101954 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:39.101959 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:39.102193 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:39.102939 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:39.102946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:39.102951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:39.103501 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:39.103820 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:39.104655 (+ 835us) server_negotiation.cc:299] Negotiation successful
0927 18:43:39.104943 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:43:39.119556 22966 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:39.121815 22966 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:39.136981 22966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f8b5609015a4f46863b8a617a47c94b: Generated new TSK 0
I20260927 18:43:39.138546 22966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:39.151823 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 22882
I20260927 18:43:39.153914 22944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:39.255201 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 22882
W20260927 18:43:39.273699 22867 connection.cc:535] client connection to 127.0.0.1:45793 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45793): Connection reset by peer (error 104)
2026-09-27T18:43:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2122 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:39 dist-test-slave-9jll krb5kdc[22981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:39 dist-test-slave-9jll krb5kdc[22981](info): set up 2 sockets
Sep 27 18:43:39 dist-test-slave-9jll krb5kdc[22981](info): commencing operation
krb5kdc: starting...
W20260927 18:43:40.310285 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.006s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:40 dist-test-slave-9jll krb5kdc[22981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:43:40.618451 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:41.009790 22994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:41.010213 22994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:41.010378 22994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:41.018440 22994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:41.018596 22994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:41.018682 22994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:41.018769 22994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:41.018842 22994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:41.030048 22994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43473
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:41.031773 22994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:41.033358 22994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:41.046316 22999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:41.046279 23002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:41.046279 23000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:41.047277 22994 server_base.cc:1034] running on GCE node
I20260927 18:43:41.048204 22994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:41.050107 22994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:41.051331 22994 hybrid_clock.cc:648] HybridClock initialized: now 1790534621051275 us; error 43 us; skew 500 ppm
I20260927 18:43:41.051733 22994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:41 dist-test-slave-9jll krb5kdc[22981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:41.055413 22994 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:43:41.057367 22994 webserver.cc:460] Webserver started at http://127.0.0.1:40607/ using document root <none> and password file <none>
I20260927 18:43:41.058286 22994 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:41.058419 22994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:41.058801 22994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:41.062788 22994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "e9d2bb944e424bf1ae5360bae59fa21c"
format_stamp: "Formatted at 2026-09-27 18:43:41 on dist-test-slave-9jll"
server_key: "609cf3c63e8e667c9a0574ec6c8ebdfd"
server_key_iv: "b348fac7e6842e618867a8e8bfcd3477"
server_key_version: "encryptionkey@0"
I20260927 18:43:41.063725 22994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "e9d2bb944e424bf1ae5360bae59fa21c"
format_stamp: "Formatted at 2026-09-27 18:43:41 on dist-test-slave-9jll"
server_key: "609cf3c63e8e667c9a0574ec6c8ebdfd"
server_key_iv: "b348fac7e6842e618867a8e8bfcd3477"
server_key_version: "encryptionkey@0"
I20260927 18:43:41.069944 22994 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 18:43:41.073972 23009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:41.075536 22994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:43:41.075703 22994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "e9d2bb944e424bf1ae5360bae59fa21c"
format_stamp: "Formatted at 2026-09-27 18:43:41 on dist-test-slave-9jll"
server_key: "609cf3c63e8e667c9a0574ec6c8ebdfd"
server_key_iv: "b348fac7e6842e618867a8e8bfcd3477"
server_key_version: "encryptionkey@0"
I20260927 18:43:41.075935 22994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:41.102840 22994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:41.107843 22994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:41.108183 22994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:41.128690 22994 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43473
I20260927 18:43:41.128702 23060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43473 every 8 connection(s)
I20260927 18:43:41.130465 22994 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:41.135123 23061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:41.140676 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 22994
I20260927 18:43:41.140857 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:41.141347 2429 external_mini_cluster.cc:1428] Setting key 4ab6d9ec14a44c56b02f5ec646a497d7
I20260927 18:43:41.149128 23061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c: Bootstrap starting.
I20260927 18:43:41.153589 23061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:41.155117 23061 log.cc:824] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:41.158212 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:41.143455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58061 (local address 127.0.0.1:43473)
0927 18:43:41.144377 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:41.144394 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:41.144456 (+ 62us) server_negotiation.cc:407] Connection header received
0927 18:43:41.147085 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:41.147120 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:41.147716 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:41.148187 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:41.148776 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:41.150904 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:41.152298 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:41.153370 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:41.153603 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:41.154918 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:41.154930 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:41.154953 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:41.155827 (+ 874us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:41.155962 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:41.156003 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:41.156165 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:41.156495 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 18:43:41.156710 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":384,"threads_started":1}
I20260927 18:43:41.159032 23061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c: No bootstrap required, opened a new log
I20260927 18:43:41.162420 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:43:41.167608 23061 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d2bb944e424bf1ae5360bae59fa21c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43473 } }
I20260927 18:43:41.168206 23061 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:41.168334 23061 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d2bb944e424bf1ae5360bae59fa21c, State: Initialized, Role: FOLLOWER
I20260927 18:43:41.169111 23061 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d2bb944e424bf1ae5360bae59fa21c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43473 } }
I20260927 18:43:41.169371 23061 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:41.169517 23061 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:41.169739 23061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:41.173223 23061 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d2bb944e424bf1ae5360bae59fa21c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43473 } }
I20260927 18:43:41.174044 23061 leader_election.cc:302] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9d2bb944e424bf1ae5360bae59fa21c; no voters:
I20260927 18:43:41.174667 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:41.167151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50585 (local address 127.0.0.1:43473)
0927 18:43:41.167390 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:41.167396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:41.167760 (+ 364us) server_negotiation.cc:407] Connection header received
0927 18:43:41.167963 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:41.167967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:41.168041 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:41.168129 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:41.168823 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:41.169762 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:41.171718 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:41.172210 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:41.172312 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:41.173797 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:41.173803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:41.173806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:41.173829 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:41.173896 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:41.173914 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:41.174002 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:41.174290 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 18:43:41.174367 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:43:41.174913 23061 leader_election.cc:288] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:41.175024 23071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:41.177253 23071 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [term 1 LEADER]: Becoming Leader. State: Replica: e9d2bb944e424bf1ae5360bae59fa21c, State: Running, Role: LEADER
I20260927 18:43:41.178004 23071 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d2bb944e424bf1ae5360bae59fa21c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43473 } }
I20260927 18:43:41.178555 23061 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:41.182484 23072 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9d2bb944e424bf1ae5360bae59fa21c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d2bb944e424bf1ae5360bae59fa21c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43473 } } }
I20260927 18:43:41.182862 23072 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:41.183391 23073 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9d2bb944e424bf1ae5360bae59fa21c. Latest consensus state: current_term: 1 leader_uuid: "e9d2bb944e424bf1ae5360bae59fa21c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d2bb944e424bf1ae5360bae59fa21c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43473 } } }
I20260927 18:43:41.183681 23073 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:41.183795 23077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:41.190454 23077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:41.201860 23077 catalog_manager.cc:1347] Generated new cluster ID: 28b76dcedddd402aba97102fec8fa484
I20260927 18:43:41.202018 23077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:41.235687 23077 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:41.237278 23077 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:41.252671 23077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9d2bb944e424bf1ae5360bae59fa21c: Generated new TSK 0
I20260927 18:43:41.253820 23077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:41.265054 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 22994
W20260927 18:43:41.266556 23056 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58061
W20260927 18:43:41.267465 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 22994: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:43:41.267674 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 22994
W20260927 18:43:41.285301 22979 connection.cc:535] client connection to 127.0.0.1:43473 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43473): Connection reset by peer (error 104)
2026-09-27T18:43:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:41 dist-test-slave-9jll krb5kdc[23092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:41 dist-test-slave-9jll krb5kdc[23092](info): set up 2 sockets
Sep 27 18:43:41 dist-test-slave-9jll krb5kdc[23092](info): commencing operation
krb5kdc: starting...
W20260927 18:43:42.314152 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:42 dist-test-slave-9jll krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:43:42.622166 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:43.029975 23105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:43.030381 23105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:43.030529 23105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:43.038609 23105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:43.038755 23105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:43.038838 23105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:43.038925 23105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:43.038985 23105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:43.050550 23105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:43.052384 23105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:43.053994 23105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:43.068011 23110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:43.068049 23111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:43.068015 23113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:43.069006 23105 server_base.cc:1034] running on GCE node
I20260927 18:43:43.070075 23105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:43.071946 23105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:43.073210 23105 hybrid_clock.cc:648] HybridClock initialized: now 1790534623073157 us; error 39 us; skew 500 ppm
I20260927 18:43:43.073637 23105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:43 dist-test-slave-9jll krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:43.077327 23105 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:43:43.079332 23105 webserver.cc:460] Webserver started at http://127.0.0.1:46491/ using document root <none> and password file <none>
I20260927 18:43:43.080202 23105 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:43.080336 23105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:43.080691 23105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:43.084331 23105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "629c2dbe82ff40fea6a70540063a20ca"
format_stamp: "Formatted at 2026-09-27 18:43:43 on dist-test-slave-9jll"
server_key: "113f5bff24097ad31737fc2265d9b6c7"
server_key_iv: "a25e1b5e6c7b017903b44cba7895b4a7"
server_key_version: "encryptionkey@0"
I20260927 18:43:43.085206 23105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "629c2dbe82ff40fea6a70540063a20ca"
format_stamp: "Formatted at 2026-09-27 18:43:43 on dist-test-slave-9jll"
server_key: "113f5bff24097ad31737fc2265d9b6c7"
server_key_iv: "a25e1b5e6c7b017903b44cba7895b4a7"
server_key_version: "encryptionkey@0"
I20260927 18:43:43.091579 23105 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 18:43:43.095980 23120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:43.097733 23105 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:43:43.097913 23105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "629c2dbe82ff40fea6a70540063a20ca"
format_stamp: "Formatted at 2026-09-27 18:43:43 on dist-test-slave-9jll"
server_key: "113f5bff24097ad31737fc2265d9b6c7"
server_key_iv: "a25e1b5e6c7b017903b44cba7895b4a7"
server_key_version: "encryptionkey@0"
I20260927 18:43:43.098117 23105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:43.104633 23105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:43.109635 23105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:43.109992 23105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:43.129863 23105 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39491
I20260927 18:43:43.129895 23171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39491 every 8 connection(s)
I20260927 18:43:43.131620 23105 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:43.136369 23172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:43.136680 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 23105
I20260927 18:43:43.136888 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:43.137528 2429 external_mini_cluster.cc:1428] Setting key 3b1571d50e2350f93d1dd6084ff39ced
I20260927 18:43:43.148342 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:43.139902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58803 (local address 127.0.0.1:39491)
0927 18:43:43.140971 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:43.140990 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:43.141383 (+ 393us) server_negotiation.cc:407] Connection header received
0927 18:43:43.143657 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:43.143691 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:43.144377 (+ 686us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:43.144958 (+ 581us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:43.146103 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:43.146112 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:43.146141 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:43.146160 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:43.146236 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:43.146271 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:43.146504 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:43.146653 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 18:43:43.146874 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":431,"threads_started":1}
I20260927 18:43:43.152598 23172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca: Bootstrap starting.
I20260927 18:43:43.152829 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:43:43.156376 23172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:43.157792 23172 log.cc:824] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:43.160899 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:43.157910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36423 (local address 127.0.0.1:39491)
0927 18:43:43.158139 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:43.158146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:43.158689 (+ 543us) server_negotiation.cc:407] Connection header received
0927 18:43:43.158897 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:43.158904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:43.158970 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:43.159074 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:43.160218 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:43.160223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:43.160226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:43.160241 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:43.160287 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:43.160302 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:43.160411 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:43.160531 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 18:43:43.160602 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:43:43.161605 23172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca: No bootstrap required, opened a new log
I20260927 18:43:43.170135 23172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629c2dbe82ff40fea6a70540063a20ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } }
I20260927 18:43:43.170650 23172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:43.170753 23172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 629c2dbe82ff40fea6a70540063a20ca, State: Initialized, Role: FOLLOWER
I20260927 18:43:43.171451 23172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629c2dbe82ff40fea6a70540063a20ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } }
I20260927 18:43:43.171707 23172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:43.171854 23172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:43.172015 23172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:43.174733 23172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629c2dbe82ff40fea6a70540063a20ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } }
I20260927 18:43:43.175348 23172 leader_election.cc:302] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 629c2dbe82ff40fea6a70540063a20ca; no voters:
I20260927 18:43:43.176028 23172 leader_election.cc:288] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:43.176186 23182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:43.176970 23182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [term 1 LEADER]: Becoming Leader. State: Replica: 629c2dbe82ff40fea6a70540063a20ca, State: Running, Role: LEADER
I20260927 18:43:43.177600 23182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629c2dbe82ff40fea6a70540063a20ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } }
I20260927 18:43:43.178545 23172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:43.180706 23184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 629c2dbe82ff40fea6a70540063a20ca. Latest consensus state: current_term: 1 leader_uuid: "629c2dbe82ff40fea6a70540063a20ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629c2dbe82ff40fea6a70540063a20ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } } }
I20260927 18:43:43.181931 23184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:43.182597 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "629c2dbe82ff40fea6a70540063a20ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "629c2dbe82ff40fea6a70540063a20ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39491 } } }
I20260927 18:43:43.182896 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:43.184265 23190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:43.190954 23190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:43.202416 23190 catalog_manager.cc:1347] Generated new cluster ID: 89c9eda1446441e8a27ad8467449a43c
I20260927 18:43:43.202586 23190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:43.234182 23190 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:43.234795 23190 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:43.247960 23190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 629c2dbe82ff40fea6a70540063a20ca: Generated new TSK 0
I20260927 18:43:43.249243 23190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:43.267462 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23105
W20260927 18:43:43.268986 23167 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58803
W20260927 18:43:43.269788 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23105: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:43:43.269919 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 23105
2026-09-27T18:43:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:43 dist-test-slave-9jll krb5kdc[23203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:43 dist-test-slave-9jll krb5kdc[23203](info): set up 2 sockets
Sep 27 18:43:43 dist-test-slave-9jll krb5kdc[23203](info): commencing operation
krb5kdc: starting...
W20260927 18:43:44.327620 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:44 dist-test-slave-9jll krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:43:44.642462 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:45.038364 23216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:45.038848 23216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:45.039016 23216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:45.047399 23216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:45.047559 23216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:45.047619 23216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:45.047722 23216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:45.047787 23216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:45.059517 23216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:45.061271 23216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:45.063041 23216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:45.076375 23221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:45.076300 23222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:45.076321 23224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:45.077081 23216 server_base.cc:1034] running on GCE node
I20260927 18:43:45.078123 23216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:45.079869 23216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:45.081087 23216 hybrid_clock.cc:648] HybridClock initialized: now 1790534625081027 us; error 58 us; skew 500 ppm
I20260927 18:43:45.081486 23216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:45 dist-test-slave-9jll krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:45.085086 23216 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:43:45.087150 23216 webserver.cc:460] Webserver started at http://127.0.0.1:42177/ using document root <none> and password file <none>
I20260927 18:43:45.088054 23216 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:45.088192 23216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:45.088546 23216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:45.092279 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "c56808a1f3e741de9ba9983e65149354"
format_stamp: "Formatted at 2026-09-27 18:43:45 on dist-test-slave-9jll"
server_key: "1c56856a16605eb2d0ad6736e903bca0"
server_key_iv: "ad906d40fafa66520529eb7cfb238d13"
server_key_version: "encryptionkey@0"
I20260927 18:43:45.093146 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "c56808a1f3e741de9ba9983e65149354"
format_stamp: "Formatted at 2026-09-27 18:43:45 on dist-test-slave-9jll"
server_key: "1c56856a16605eb2d0ad6736e903bca0"
server_key_iv: "ad906d40fafa66520529eb7cfb238d13"
server_key_version: "encryptionkey@0"
I20260927 18:43:45.099503 23216 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 18:43:45.103619 23231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:45.105141 23216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 18:43:45.105337 23216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "c56808a1f3e741de9ba9983e65149354"
format_stamp: "Formatted at 2026-09-27 18:43:45 on dist-test-slave-9jll"
server_key: "1c56856a16605eb2d0ad6736e903bca0"
server_key_iv: "ad906d40fafa66520529eb7cfb238d13"
server_key_version: "encryptionkey@0"
I20260927 18:43:45.105545 23216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:45.129828 23216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:45.134878 23216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:45.135200 23216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:45.155385 23216 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33611
I20260927 18:43:45.155439 23282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33611 every 8 connection(s)
I20260927 18:43:45.157097 23216 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:45.161690 23283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:45.165051 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 23216
I20260927 18:43:45.165262 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:45.165918 2429 external_mini_cluster.cc:1428] Setting key 367caf403c4a7498fa874d1cc329968a
I20260927 18:43:45.175884 23283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354: Bootstrap starting.
I20260927 18:43:45.180581 23283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:45.182229 23283 log.cc:824] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:43:45 dist-test-slave-9jll krb5kdc[23203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:43:45.185794 23283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354: No bootstrap required, opened a new log
I20260927 18:43:45.192158 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:45.168309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60883 (local address 127.0.0.1:33611)
0927 18:43:45.169206 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:45.169221 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:45.169273 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:43:45.171238 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:45.171267 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:45.171777 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:45.172216 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:45.174251 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:45.175707 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:45.177532 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:45.178470 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:45.178616 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:45.183910 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:45.183920 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:45.183939 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:45.183978 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:45.187222 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:45.188210 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:45.188220 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:45.188229 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:45.188424 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:45.188854 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:45.188857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:45.188859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:45.189533 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:45.189911 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:45.190335 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 18:43:45.190836 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":412,"threads_started":1}
I20260927 18:43:45.195494 23283 raft_consensus.cc:348] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56808a1f3e741de9ba9983e65149354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } }
I20260927 18:43:45.195963 23283 raft_consensus.cc:374] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:45.196066 23283 raft_consensus.cc:729] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c56808a1f3e741de9ba9983e65149354, State: Initialized, Role: FOLLOWER
I20260927 18:43:45.196645 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:43:45.196805 23283 consensus_queue.cc:260] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56808a1f3e741de9ba9983e65149354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } }
I20260927 18:43:45.197095 23283 raft_consensus.cc:388] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:45.197232 23283 raft_consensus.cc:482] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:45.197412 23283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:45.201083 23283 raft_consensus.cc:504] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56808a1f3e741de9ba9983e65149354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } }
I20260927 18:43:45.201982 23283 leader_election.cc:302] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c56808a1f3e741de9ba9983e65149354; no voters:
I20260927 18:43:45.202956 23283 leader_election.cc:288] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:45.203109 23292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:45.204372 23292 raft_consensus.cc:686] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [term 1 LEADER]: Becoming Leader. State: Replica: c56808a1f3e741de9ba9983e65149354, State: Running, Role: LEADER
I20260927 18:43:45.205185 23292 consensus_queue.cc:237] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56808a1f3e741de9ba9983e65149354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } }
I20260927 18:43:45.207115 23283 sys_catalog.cc:564] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:45.207907 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c56808a1f3e741de9ba9983e65149354. Latest consensus state: current_term: 1 leader_uuid: "c56808a1f3e741de9ba9983e65149354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56808a1f3e741de9ba9983e65149354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } } }
I20260927 18:43:45.208240 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:45.208804 23294 sys_catalog.cc:455] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c56808a1f3e741de9ba9983e65149354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56808a1f3e741de9ba9983e65149354" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } } }
I20260927 18:43:45.209183 23294 sys_catalog.cc:458] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:45.220930 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:45.202967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48229 (local address 127.0.0.1:33611)
0927 18:43:45.203219 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:45.203227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:45.203905 (+ 678us) server_negotiation.cc:407] Connection header received
0927 18:43:45.204119 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:45.204125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:45.204250 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:45.204381 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:45.206814 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:45.207984 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:45.209950 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:45.210776 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:45.210894 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:45.214511 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:45.214520 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:45.214527 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:45.214583 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:45.218198 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:45.218893 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:45.218897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:45.218900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:45.219025 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:45.219421 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:45.219425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:45.219427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:45.219750 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:45.219988 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:45.220342 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 18:43:45.220573 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 18:43:45.222471 23298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:45.228039 23298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:45.238176 23298 catalog_manager.cc:1347] Generated new cluster ID: 0d9f8a3d1f3b4f2eb92b923d4d1a93d0
I20260927 18:43:45.238327 23298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:45.272403 23298 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:45.273959 23298 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:45.289228 23298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c56808a1f3e741de9ba9983e65149354: Generated new TSK 0
I20260927 18:43:45.290627 23298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:45.305104 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23216
I20260927 18:43:45.306279 23278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:45.413015 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 23216
2026-09-27T18:43:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2144 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:45 dist-test-slave-9jll krb5kdc[23315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:45 dist-test-slave-9jll krb5kdc[23315](info): set up 2 sockets
Sep 27 18:43:45 dist-test-slave-9jll krb5kdc[23315](info): commencing operation
krb5kdc: starting...
W20260927 18:43:46.470446 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:46 dist-test-slave-9jll krb5kdc[23315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:43:46.782692 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:47.172468 23328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:47.172854 23328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:47.172986 23328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:47.180874 23328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:47.181023 23328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:47.181078 23328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:47.181173 23328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:47.181231 23328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:47.192195 23328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:47.193926 23328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:47.195554 23328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:47.208787 23333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:47.208829 23334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:47.208899 23336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:47.209753 23328 server_base.cc:1034] running on GCE node
I20260927 18:43:47.210667 23328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:47.212407 23328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:47.213603 23328 hybrid_clock.cc:648] HybridClock initialized: now 1790534627213519 us; error 77 us; skew 500 ppm
I20260927 18:43:47.214023 23328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:47 dist-test-slave-9jll krb5kdc[23315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:47.217518 23328 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:43:47.219290 23328 webserver.cc:460] Webserver started at http://127.0.0.1:40773/ using document root <none> and password file <none>
I20260927 18:43:47.220096 23328 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:47.220202 23328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:47.220505 23328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:47.223991 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "839b598395b34102baea5afcac8bc219"
format_stamp: "Formatted at 2026-09-27 18:43:47 on dist-test-slave-9jll"
server_key: "1ffbcb4ab7349230504ba8f985cd80d8"
server_key_iv: "c2bfe128da73f41a5b0dfc37661b6fd7"
server_key_version: "encryptionkey@0"
I20260927 18:43:47.224788 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "839b598395b34102baea5afcac8bc219"
format_stamp: "Formatted at 2026-09-27 18:43:47 on dist-test-slave-9jll"
server_key: "1ffbcb4ab7349230504ba8f985cd80d8"
server_key_iv: "c2bfe128da73f41a5b0dfc37661b6fd7"
server_key_version: "encryptionkey@0"
I20260927 18:43:47.230901 23328 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 18:43:47.235276 23343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:47.236975 23328 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 18:43:47.237176 23328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "839b598395b34102baea5afcac8bc219"
format_stamp: "Formatted at 2026-09-27 18:43:47 on dist-test-slave-9jll"
server_key: "1ffbcb4ab7349230504ba8f985cd80d8"
server_key_iv: "c2bfe128da73f41a5b0dfc37661b6fd7"
server_key_version: "encryptionkey@0"
I20260927 18:43:47.237417 23328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:47.257933 23328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:47.262904 23328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:47.263217 23328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:47.283562 23328 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44129
I20260927 18:43:47.283583 23394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44129 every 8 connection(s)
I20260927 18:43:47.285193 23328 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:47.289943 23395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:47.295392 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 23328
I20260927 18:43:47.295576 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:47.296077 2429 external_mini_cluster.cc:1428] Setting key 35d1e1609d1eb81a7a6182d3afe7aaf2
I20260927 18:43:47.303874 23395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219: Bootstrap starting.
I20260927 18:43:47.308218 23395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:47.309698 23395 log.cc:824] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:47.312578 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:47.298320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36139 (local address 127.0.0.1:44129)
0927 18:43:47.299256 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:47.299274 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:47.299339 (+ 65us) server_negotiation.cc:407] Connection header received
0927 18:43:47.301969 (+ 2630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:47.302002 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:47.302622 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:47.303104 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:47.303684 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:47.305765 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:47.307176 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:47.308266 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:47.308513 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:47.309713 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:47.309726 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:47.309747 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:47.310618 (+ 871us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:47.310709 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:47.310735 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:47.310829 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:47.311172 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 18:43:47.311455 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":373,"threads_started":1}
I20260927 18:43:47.313249 23395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219: No bootstrap required, opened a new log
I20260927 18:43:47.316998 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:43:47.324303 23395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839b598395b34102baea5afcac8bc219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } }
I20260927 18:43:47.324990 23395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:47.325146 23395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839b598395b34102baea5afcac8bc219, State: Initialized, Role: FOLLOWER
I20260927 18:43:47.326090 23395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839b598395b34102baea5afcac8bc219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } }
I20260927 18:43:47.326502 23395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:47.326731 23395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:47.326988 23395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:47.330749 23395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839b598395b34102baea5afcac8bc219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } }
I20260927 18:43:47.331502 23395 leader_election.cc:302] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 839b598395b34102baea5afcac8bc219; no voters:
I20260927 18:43:47.332538 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:47.324431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51493 (local address 127.0.0.1:44129)
0927 18:43:47.324929 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:47.324936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:47.324955 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:43:47.325043 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:47.325049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:47.325144 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:47.325252 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:47.326022 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:47.327361 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:47.329302 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:47.330001 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:47.330118 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:47.331616 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:47.331622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:47.331627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:47.331652 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:47.331705 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:47.331730 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:47.331816 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:47.332056 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 18:43:47.332162 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260927 18:43:47.332600 23395 leader_election.cc:288] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:47.332719 23405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:47.336349 23405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [term 1 LEADER]: Becoming Leader. State: Replica: 839b598395b34102baea5afcac8bc219, State: Running, Role: LEADER
I20260927 18:43:47.337323 23405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839b598395b34102baea5afcac8bc219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } }
I20260927 18:43:47.338719 23395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:47.340937 23406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 839b598395b34102baea5afcac8bc219. Latest consensus state: current_term: 1 leader_uuid: "839b598395b34102baea5afcac8bc219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839b598395b34102baea5afcac8bc219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } } }
I20260927 18:43:47.341375 23406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:47.341981 23407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "839b598395b34102baea5afcac8bc219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839b598395b34102baea5afcac8bc219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44129 } } }
I20260927 18:43:47.342296 23407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:47.350337 23410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:47.356877 23410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:47.368508 23410 catalog_manager.cc:1347] Generated new cluster ID: 621ec686ef1b4ba7a1b8f73ce89fd70c
I20260927 18:43:47.368659 23410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:47.387171 23410 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:47.389377 23410 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:47.400235 23410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 839b598395b34102baea5afcac8bc219: Generated new TSK 0
I20260927 18:43:47.401768 23410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:47.411387 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23328
W20260927 18:43:47.412793 23390 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36139
W20260927 18:43:47.413575 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23328: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:43:47.413751 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 23328
2026-09-27T18:43:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:47 dist-test-slave-9jll krb5kdc[23426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:47 dist-test-slave-9jll krb5kdc[23426](info): set up 2 sockets
Sep 27 18:43:47 dist-test-slave-9jll krb5kdc[23426](info): commencing operation
krb5kdc: starting...
W20260927 18:43:48.466589 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:48 dist-test-slave-9jll krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:43:48.773407 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:49.163056 23439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:49.163441 23439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:49.163590 23439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:49.171764 23439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:49.171949 23439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:49.172041 23439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:49.172128 23439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:49.172189 23439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:49.183487 23439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:49.185217 23439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:49.186872 23439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:49.200655 23444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:49.200673 23447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:49.200717 23445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:49.201790 23439 server_base.cc:1034] running on GCE node
I20260927 18:43:49.202767 23439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:49.204591 23439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:49.205827 23439 hybrid_clock.cc:648] HybridClock initialized: now 1790534629205778 us; error 39 us; skew 500 ppm
I20260927 18:43:49.206261 23439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:49 dist-test-slave-9jll krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:49.219424 23439 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:43:49.221544 23439 webserver.cc:460] Webserver started at http://127.0.0.1:44005/ using document root <none> and password file <none>
I20260927 18:43:49.222461 23439 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:49.222620 23439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:49.222970 23439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:49.226547 23439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "c4c55fc0f300430faf463e6b729ed2ee"
format_stamp: "Formatted at 2026-09-27 18:43:49 on dist-test-slave-9jll"
server_key: "3c4f64dcfbb92821c4b65ee22df3eda6"
server_key_iv: "5d83667b8916d3f40a8b56629e711177"
server_key_version: "encryptionkey@0"
I20260927 18:43:49.227424 23439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "c4c55fc0f300430faf463e6b729ed2ee"
format_stamp: "Formatted at 2026-09-27 18:43:49 on dist-test-slave-9jll"
server_key: "3c4f64dcfbb92821c4b65ee22df3eda6"
server_key_iv: "5d83667b8916d3f40a8b56629e711177"
server_key_version: "encryptionkey@0"
I20260927 18:43:49.233556 23439 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:43:49.237542 23454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:49.239167 23439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:43:49.239358 23439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "c4c55fc0f300430faf463e6b729ed2ee"
format_stamp: "Formatted at 2026-09-27 18:43:49 on dist-test-slave-9jll"
server_key: "3c4f64dcfbb92821c4b65ee22df3eda6"
server_key_iv: "5d83667b8916d3f40a8b56629e711177"
server_key_version: "encryptionkey@0"
I20260927 18:43:49.239573 23439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:49.246446 23439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:49.251232 23439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:49.251542 23439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:49.271665 23439 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37897
I20260927 18:43:49.271734 23505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37897 every 8 connection(s)
I20260927 18:43:49.273504 23439 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:49.275310 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 23439
I20260927 18:43:49.275504 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:49.276010 2429 external_mini_cluster.cc:1428] Setting key 16654ef6d193020bee9c74c807d9c78c
I20260927 18:43:49.278995 23506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:49.285650 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:49.278214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33747 (local address 127.0.0.1:37897)
0927 18:43:49.279158 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:49.279174 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:49.279226 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:43:49.281106 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:49.281131 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:49.281550 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:49.282015 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:49.283531 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:49.283538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:49.283554 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:49.283574 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:49.283676 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:49.283718 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:49.283935 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:49.284054 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 18:43:49.284314 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":389,"threads_started":1}
I20260927 18:43:49.290338 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:43:49.293138 23506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee: Bootstrap starting.
I20260927 18:43:49.297624 23506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:49.298105 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:49.295412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54853 (local address 127.0.0.1:37897)
0927 18:43:49.295621 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:49.295629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:49.296000 (+ 371us) server_negotiation.cc:407] Connection header received
0927 18:43:49.296197 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:49.296201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:49.296266 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:49.296361 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:49.297380 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:49.297384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:49.297388 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:49.297405 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:49.297453 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:49.297470 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:49.297580 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:49.297726 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 18:43:49.297816 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:43:49.299134 23506 log.cc:824] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:49.302465 23506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee: No bootstrap required, opened a new log
I20260927 18:43:49.311792 23506 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c55fc0f300430faf463e6b729ed2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37897 } }
I20260927 18:43:49.312386 23506 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:49.312495 23506 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c55fc0f300430faf463e6b729ed2ee, State: Initialized, Role: FOLLOWER
I20260927 18:43:49.313344 23506 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c55fc0f300430faf463e6b729ed2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37897 } }
I20260927 18:43:49.313694 23506 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:49.313876 23506 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:49.314064 23506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:49.317512 23506 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c55fc0f300430faf463e6b729ed2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37897 } }
I20260927 18:43:49.318254 23506 leader_election.cc:302] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4c55fc0f300430faf463e6b729ed2ee; no voters:
I20260927 18:43:49.318979 23506 leader_election.cc:288] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:49.319130 23516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:49.319823 23516 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [term 1 LEADER]: Becoming Leader. State: Replica: c4c55fc0f300430faf463e6b729ed2ee, State: Running, Role: LEADER
I20260927 18:43:49.320457 23516 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c55fc0f300430faf463e6b729ed2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37897 } }
I20260927 18:43:49.321509 23506 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:49.324028 23517 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4c55fc0f300430faf463e6b729ed2ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c55fc0f300430faf463e6b729ed2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37897 } } }
I20260927 18:43:49.323974 23518 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4c55fc0f300430faf463e6b729ed2ee. Latest consensus state: current_term: 1 leader_uuid: "c4c55fc0f300430faf463e6b729ed2ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c55fc0f300430faf463e6b729ed2ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37897 } } }
I20260927 18:43:49.324509 23517 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:49.324564 23518 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:49.325618 23522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:49.333628 23522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:49.345459 23522 catalog_manager.cc:1347] Generated new cluster ID: 41f3b38f6474486493005043b5ab1c09
I20260927 18:43:49.345716 23522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:49.370350 23522 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:49.370949 23522 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:49.388679 23522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4c55fc0f300430faf463e6b729ed2ee: Generated new TSK 0
I20260927 18:43:49.389935 23522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:49.409725 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23439
W20260927 18:43:49.411126 23501 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33747
W20260927 18:43:49.411903 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23439: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:43:49.412043 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 23439
2026-09-27T18:43:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:49 dist-test-slave-9jll krb5kdc[23537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:49 dist-test-slave-9jll krb5kdc[23537](info): set up 2 sockets
Sep 27 18:43:49 dist-test-slave-9jll krb5kdc[23537](info): commencing operation
krb5kdc: starting...
W20260927 18:43:50.466509 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:50 dist-test-slave-9jll krb5kdc[23537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:43:50.785362 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:51.181453 23550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:51.181891 23550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:51.182046 23550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:51.190028 23550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:51.190197 23550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:51.190292 23550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:51.190398 23550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:51.190464 23550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:51.201812 23550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:51.203572 23550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:51.205271 23550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:51.219161 23555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:51.219148 23556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:51.219086 23558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:51.220640 23550 server_base.cc:1034] running on GCE node
I20260927 18:43:51.221611 23550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:51.223424 23550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:51.224651 23550 hybrid_clock.cc:648] HybridClock initialized: now 1790534631224606 us; error 37 us; skew 500 ppm
I20260927 18:43:51.225044 23550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:51 dist-test-slave-9jll krb5kdc[23537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:51.228641 23550 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:43:51.230471 23550 webserver.cc:460] Webserver started at http://127.0.0.1:39483/ using document root <none> and password file <none>
I20260927 18:43:51.231351 23550 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:51.231479 23550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:51.231834 23550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:51.235618 23550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "6cc6ac5e995643caa1132897da975072"
format_stamp: "Formatted at 2026-09-27 18:43:51 on dist-test-slave-9jll"
server_key: "ba2a6088b145d7710e111dafc14da204"
server_key_iv: "1150d392c2a74400957871ed38fb16bb"
server_key_version: "encryptionkey@0"
I20260927 18:43:51.236565 23550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "6cc6ac5e995643caa1132897da975072"
format_stamp: "Formatted at 2026-09-27 18:43:51 on dist-test-slave-9jll"
server_key: "ba2a6088b145d7710e111dafc14da204"
server_key_iv: "1150d392c2a74400957871ed38fb16bb"
server_key_version: "encryptionkey@0"
I20260927 18:43:51.243561 23550 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260927 18:43:51.247922 23565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:51.249758 23550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:43:51.249971 23550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "6cc6ac5e995643caa1132897da975072"
format_stamp: "Formatted at 2026-09-27 18:43:51 on dist-test-slave-9jll"
server_key: "ba2a6088b145d7710e111dafc14da204"
server_key_iv: "1150d392c2a74400957871ed38fb16bb"
server_key_version: "encryptionkey@0"
I20260927 18:43:51.250212 23550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:51.267058 23550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:51.272475 23550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:51.272794 23550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:51.293013 23550 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33179
I20260927 18:43:51.293033 23616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33179 every 8 connection(s)
I20260927 18:43:51.294709 23550 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:51.298091 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 23550
I20260927 18:43:51.298303 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:51.298825 2429 external_mini_cluster.cc:1428] Setting key 90004aa29b6ffd5b243b3785eb67882e
I20260927 18:43:51.299695 23617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 18:43:51 dist-test-slave-9jll krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:43:51.315976 23617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072: Bootstrap starting.
I20260927 18:43:51.320138 23617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:51.321743 23617 log.cc:824] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:51.325517 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:51.300809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:53255 (local address 127.0.0.1:33179)
0927 18:43:51.301605 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:51.301619 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:51.301720 (+ 101us) server_negotiation.cc:407] Connection header received
0927 18:43:51.303470 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:51.303495 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:51.303993 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:51.304382 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:51.306453 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:51.308088 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:51.309952 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:51.310849 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:51.311046 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:51.316419 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:51.316430 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:51.316450 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:51.316502 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:51.319787 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:51.320965 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:51.320973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:51.320981 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:51.321201 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:51.321775 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:51.321779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:51.321782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:51.322497 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:51.322792 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:51.323520 (+ 728us) server_negotiation.cc:299] Negotiation successful
0927 18:43:51.323949 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":342,"threads_started":1}
I20260927 18:43:51.326035 23617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072: No bootstrap required, opened a new log
I20260927 18:43:51.330206 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:43:51.335691 23617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc6ac5e995643caa1132897da975072" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33179 } }
I20260927 18:43:51.336251 23617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:51.336359 23617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc6ac5e995643caa1132897da975072, State: Initialized, Role: FOLLOWER
I20260927 18:43:51.337100 23617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc6ac5e995643caa1132897da975072" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33179 } }
I20260927 18:43:51.337361 23617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:51.337502 23617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:51.337728 23617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:51.340335 23617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc6ac5e995643caa1132897da975072" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33179 } }
I20260927 18:43:51.340991 23617 leader_election.cc:302] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cc6ac5e995643caa1132897da975072; no voters:
I20260927 18:43:51.341743 23617 leader_election.cc:288] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:51.341883 23627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:51.344118 23627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [term 1 LEADER]: Becoming Leader. State: Replica: 6cc6ac5e995643caa1132897da975072, State: Running, Role: LEADER
I20260927 18:43:51.344645 23617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:51.344864 23627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc6ac5e995643caa1132897da975072" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33179 } }
I20260927 18:43:51.351403 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:51.335311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:34871 (local address 127.0.0.1:33179)
0927 18:43:51.335486 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:51.335492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:51.335983 (+ 491us) server_negotiation.cc:407] Connection header received
0927 18:43:51.336166 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:51.336171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:51.336274 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:51.336383 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:51.338100 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:51.338974 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:51.340301 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:51.340837 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:51.340956 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:51.343985 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:51.343992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:51.343997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:51.344048 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:51.347867 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:51.348777 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:51.348783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:51.348787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:51.348982 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:51.349747 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:51.349752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:51.349756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:51.350085 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:51.350468 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:51.350768 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 18:43:51.350971 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:43:51.358032 23629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cc6ac5e995643caa1132897da975072. Latest consensus state: current_term: 1 leader_uuid: "6cc6ac5e995643caa1132897da975072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc6ac5e995643caa1132897da975072" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33179 } } }
I20260927 18:43:51.358026 23628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cc6ac5e995643caa1132897da975072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc6ac5e995643caa1132897da975072" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33179 } } }
I20260927 18:43:51.358665 23629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:51.358731 23628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072 [sys.catalog]: This master's current role is: LEADER
W20260927 18:43:51.362550 23643 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:43:51.362694 23643 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:43:51.362861 23644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:51.364787 23644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:51.375507 23644 catalog_manager.cc:1347] Generated new cluster ID: a081905e5ca2460dad1b19fffacceca2
I20260927 18:43:51.375661 23644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:51.399008 23644 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:51.401258 23644 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:51.417130 23644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cc6ac5e995643caa1132897da975072: Generated new TSK 0
I20260927 18:43:51.418443 23644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:51.439046 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23550
I20260927 18:43:51.440465 23612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:51.544447 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 23550
W20260927 18:43:51.562290 23535 connection.cc:535] client connection to 127.0.0.1:33179 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33179): Connection reset by peer (error 104)
2026-09-27T18:43:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2131 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:51 dist-test-slave-9jll krb5kdc[23650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:51 dist-test-slave-9jll krb5kdc[23650](info): set up 2 sockets
Sep 27 18:43:51 dist-test-slave-9jll krb5kdc[23650](info): commencing operation
krb5kdc: starting...
W20260927 18:43:52.598328 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:52 dist-test-slave-9jll krb5kdc[23650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:43:52.914127 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:53.328503 23663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:53.328941 23663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:53.329057 23663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:53.337852 23663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:53.338018 23663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:53.338078 23663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:53.338145 23663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:53.338191 23663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:53.351053 23663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:53.352936 23663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:53.354736 23663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:53.369494 23668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:53.369451 23669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:53.369421 23671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:53.370831 23663 server_base.cc:1034] running on GCE node
I20260927 18:43:53.372085 23663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:53.374558 23663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:53.375874 23663 hybrid_clock.cc:648] HybridClock initialized: now 1790534633375802 us; error 71 us; skew 500 ppm
I20260927 18:43:53.376410 23663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:53 dist-test-slave-9jll krb5kdc[23650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:53.380895 23663 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:43:53.383293 23663 webserver.cc:460] Webserver started at http://127.0.0.1:34015/ using document root <none> and password file <none>
I20260927 18:43:53.384312 23663 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:53.384470 23663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:53.384858 23663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:53.389026 23663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "ac791ba20c914e2287e1819131aa34de"
format_stamp: "Formatted at 2026-09-27 18:43:53 on dist-test-slave-9jll"
server_key: "2593d01d9c0b348261cf493f9f8dfc10"
server_key_iv: "ab6dab088a460ec268c478ae209d8fae"
server_key_version: "encryptionkey@0"
I20260927 18:43:53.390062 23663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "ac791ba20c914e2287e1819131aa34de"
format_stamp: "Formatted at 2026-09-27 18:43:53 on dist-test-slave-9jll"
server_key: "2593d01d9c0b348261cf493f9f8dfc10"
server_key_iv: "ab6dab088a460ec268c478ae209d8fae"
server_key_version: "encryptionkey@0"
I20260927 18:43:53.396587 23663 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:43:53.401510 23678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:53.403558 23663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:43:53.403780 23663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "ac791ba20c914e2287e1819131aa34de"
format_stamp: "Formatted at 2026-09-27 18:43:53 on dist-test-slave-9jll"
server_key: "2593d01d9c0b348261cf493f9f8dfc10"
server_key_iv: "ab6dab088a460ec268c478ae209d8fae"
server_key_version: "encryptionkey@0"
I20260927 18:43:53.404016 23663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:53.426137 23663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:53.432348 23663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:53.432727 23663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:53.453526 23663 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46349
I20260927 18:43:53.453543 23729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46349 every 8 connection(s)
I20260927 18:43:53.455365 23663 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:53.459259 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 23663
I20260927 18:43:53.459555 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:53.460150 2429 external_mini_cluster.cc:1428] Setting key 0fb9fa37b6211ea84be56315b5a7d63a
I20260927 18:43:53.460564 23730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:53.477154 23730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de: Bootstrap starting.
I20260927 18:43:53.477483 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:53.462518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:35383 (local address 127.0.0.1:46349)
0927 18:43:53.463486 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:53.463501 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:53.463570 (+ 69us) server_negotiation.cc:407] Connection header received
0927 18:43:53.466335 (+ 2765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:53.466367 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:53.467005 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:53.467470 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:53.468281 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:53.469879 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:53.471846 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:53.472682 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:53.472841 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:53.474524 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:53.474536 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:53.474567 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:53.475339 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:53.475446 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:53.475485 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:53.475638 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:53.475969 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 18:43:53.476244 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":425,"threads_started":1}
I20260927 18:43:53.481501 23730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:53.482122 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:43:53.482796 23730 log.cc:824] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:53.486402 23730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de: No bootstrap required, opened a new log
I20260927 18:43:53.495378 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:53.487243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:57697 (local address 127.0.0.1:46349)
0927 18:43:53.487436 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:53.487443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:53.487929 (+ 486us) server_negotiation.cc:407] Connection header received
0927 18:43:53.488129 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:53.488134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:53.488290 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:53.488410 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:53.489027 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:53.490323 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:53.491865 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:53.492637 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:53.492817 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:53.494454 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:53.494459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:53.494463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:53.494488 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:53.494540 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:53.494618 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:53.494729 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:53.494958 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 18:43:53.495035 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 18:43:53.497409 23730 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac791ba20c914e2287e1819131aa34de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } }
I20260927 18:43:53.497992 23730 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:53.498099 23730 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac791ba20c914e2287e1819131aa34de, State: Initialized, Role: FOLLOWER
I20260927 18:43:53.498839 23730 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac791ba20c914e2287e1819131aa34de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } }
I20260927 18:43:53.499110 23730 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:53.499269 23730 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:53.499464 23730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:53.502817 23730 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac791ba20c914e2287e1819131aa34de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } }
I20260927 18:43:53.503541 23730 leader_election.cc:302] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac791ba20c914e2287e1819131aa34de; no voters:
I20260927 18:43:53.504248 23730 leader_election.cc:288] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:53.504442 23740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:53.505256 23740 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [term 1 LEADER]: Becoming Leader. State: Replica: ac791ba20c914e2287e1819131aa34de, State: Running, Role: LEADER
I20260927 18:43:53.506014 23740 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac791ba20c914e2287e1819131aa34de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } }
I20260927 18:43:53.507124 23730 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:53.509315 23741 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac791ba20c914e2287e1819131aa34de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac791ba20c914e2287e1819131aa34de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } } }
I20260927 18:43:53.509861 23741 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:53.509783 23742 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac791ba20c914e2287e1819131aa34de. Latest consensus state: current_term: 1 leader_uuid: "ac791ba20c914e2287e1819131aa34de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac791ba20c914e2287e1819131aa34de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46349 } } }
I20260927 18:43:53.510056 23742 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:53.510859 23748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:53.516712 23748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:53.526901 23748 catalog_manager.cc:1347] Generated new cluster ID: f14b5bc2c2bc42ebba222392725bf6d3
I20260927 18:43:53.527055 23748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:53.537470 23748 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:53.539318 23748 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:53.562044 23748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac791ba20c914e2287e1819131aa34de: Generated new TSK 0
I20260927 18:43:53.563512 23748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:53.576125 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23663
W20260927 18:43:53.577729 23725 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.62:35383
W20260927 18:43:53.578577 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23663: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:43:53.578744 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 23663
W20260927 18:43:53.596743 23648 connection.cc:535] client connection to 127.0.0.1:46349 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46349): Connection reset by peer (error 104)
2026-09-27T18:43:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2034 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:53 dist-test-slave-9jll krb5kdc[23761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:53 dist-test-slave-9jll krb5kdc[23761](info): set up 2 sockets
Sep 27 18:43:53 dist-test-slave-9jll krb5kdc[23761](info): commencing operation
krb5kdc: starting...
W20260927 18:43:54.646332 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.004s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:54 dist-test-slave-9jll krb5kdc[23761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:43:54.961927 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:55.372470 23774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:55.372920 23774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:55.373076 23774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:55.381314 23774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:55.381485 23774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:55.381546 23774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:55.381637 23774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:55.381829 23774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:55.394165 23774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:55.395942 23774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:55.397603 23774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:55.411880 23780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:55.411921 23782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:55.411957 23779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:55.412644 23774 server_base.cc:1034] running on GCE node
I20260927 18:43:55.413854 23774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:55.415757 23774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:55.417006 23774 hybrid_clock.cc:648] HybridClock initialized: now 1790534635416954 us; error 37 us; skew 500 ppm
I20260927 18:43:55.417495 23774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:55 dist-test-slave-9jll krb5kdc[23761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:55.428360 23774 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:43:55.430526 23774 webserver.cc:460] Webserver started at http://127.0.0.1:41645/ using document root <none> and password file <none>
I20260927 18:43:55.431483 23774 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:55.431617 23774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:55.432003 23774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:55.435796 23774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "b979dac0c94f45f9a5de69d8d130b92a"
format_stamp: "Formatted at 2026-09-27 18:43:55 on dist-test-slave-9jll"
server_key: "f2ae5e1a5540273c15066012a43b6b4f"
server_key_iv: "9abe061207f28be03a53a52348eb7cca"
server_key_version: "encryptionkey@0"
I20260927 18:43:55.436731 23774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "b979dac0c94f45f9a5de69d8d130b92a"
format_stamp: "Formatted at 2026-09-27 18:43:55 on dist-test-slave-9jll"
server_key: "f2ae5e1a5540273c15066012a43b6b4f"
server_key_iv: "9abe061207f28be03a53a52348eb7cca"
server_key_version: "encryptionkey@0"
I20260927 18:43:55.443270 23774 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 18:43:55.447772 23789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:55.449697 23774 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 18:43:55.449970 23774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "b979dac0c94f45f9a5de69d8d130b92a"
format_stamp: "Formatted at 2026-09-27 18:43:55 on dist-test-slave-9jll"
server_key: "f2ae5e1a5540273c15066012a43b6b4f"
server_key_iv: "9abe061207f28be03a53a52348eb7cca"
server_key_version: "encryptionkey@0"
I20260927 18:43:55.450222 23774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:55.457100 23774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:55.462522 23774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:55.462880 23774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:55.484217 23774 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42781
I20260927 18:43:55.484270 23840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42781 every 8 connection(s)
I20260927 18:43:55.485939 23774 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:55.490748 23841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:55.495662 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 23774
I20260927 18:43:55.495867 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:55.496367 2429 external_mini_cluster.cc:1428] Setting key d88474307f6a0d163f2c4a388e114165
I20260927 18:43:55.504650 23841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a: Bootstrap starting.
I20260927 18:43:55.507050 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:55.498916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:59823 (local address 127.0.0.1:42781)
0927 18:43:55.499893 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:55.499909 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:55.499961 (+ 52us) server_negotiation.cc:407] Connection header received
0927 18:43:55.502605 (+ 2644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:55.502643 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:55.503138 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:55.503573 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:55.504996 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:55.505004 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:55.505020 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:55.505042 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:55.505127 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:55.505164 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:55.505389 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:55.505505 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 18:43:55.505797 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":438,"threads_started":1}
I20260927 18:43:55.508880 23841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:55.510533 23841 log.cc:824] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:55.512122 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:43:55.514333 23841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a: No bootstrap required, opened a new log
I20260927 18:43:55.520085 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:55.517207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:47725 (local address 127.0.0.1:42781)
0927 18:43:55.517463 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:55.517470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:55.517834 (+ 364us) server_negotiation.cc:407] Connection header received
0927 18:43:55.518032 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:55.518036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:55.518100 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:55.518190 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:55.519237 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:55.519242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:55.519246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:55.519263 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:55.519316 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:55.519336 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:55.519456 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:55.519613 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 18:43:55.519703 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 18:43:55.524279 23841 raft_consensus.cc:348] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b979dac0c94f45f9a5de69d8d130b92a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42781 } }
I20260927 18:43:55.524806 23841 raft_consensus.cc:374] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:55.524917 23841 raft_consensus.cc:729] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b979dac0c94f45f9a5de69d8d130b92a, State: Initialized, Role: FOLLOWER
I20260927 18:43:55.525830 23841 consensus_queue.cc:260] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b979dac0c94f45f9a5de69d8d130b92a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42781 } }
I20260927 18:43:55.526114 23841 raft_consensus.cc:388] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:55.526266 23841 raft_consensus.cc:482] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:55.526450 23841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:55.529114 23841 raft_consensus.cc:504] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b979dac0c94f45f9a5de69d8d130b92a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42781 } }
I20260927 18:43:55.529803 23841 leader_election.cc:302] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b979dac0c94f45f9a5de69d8d130b92a; no voters:
I20260927 18:43:55.530484 23841 leader_election.cc:288] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:55.530637 23851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:55.531370 23851 raft_consensus.cc:686] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [term 1 LEADER]: Becoming Leader. State: Replica: b979dac0c94f45f9a5de69d8d130b92a, State: Running, Role: LEADER
I20260927 18:43:55.532349 23851 consensus_queue.cc:237] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b979dac0c94f45f9a5de69d8d130b92a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42781 } }
I20260927 18:43:55.533322 23841 sys_catalog.cc:564] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:55.542188 23853 sys_catalog.cc:455] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b979dac0c94f45f9a5de69d8d130b92a. Latest consensus state: current_term: 1 leader_uuid: "b979dac0c94f45f9a5de69d8d130b92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b979dac0c94f45f9a5de69d8d130b92a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42781 } } }
I20260927 18:43:55.543879 23853 sys_catalog.cc:458] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:55.543287 23852 sys_catalog.cc:455] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b979dac0c94f45f9a5de69d8d130b92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b979dac0c94f45f9a5de69d8d130b92a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42781 } } }
I20260927 18:43:55.544811 23852 sys_catalog.cc:458] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:55.546280 23864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:55.552186 23864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:55.563530 23864 catalog_manager.cc:1347] Generated new cluster ID: 54c75ddbefc4454a9067bb6b89c3b2a8
I20260927 18:43:55.563750 23864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:55.594894 23864 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:55.595558 23864 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:55.608115 23864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b979dac0c94f45f9a5de69d8d130b92a: Generated new TSK 0
I20260927 18:43:55.609549 23864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:55.622797 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23774
W20260927 18:43:55.624349 23836 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.62:59823
W20260927 18:43:55.625267 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23774: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:43:55.625459 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 23774
2026-09-27T18:43:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2046 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:55 dist-test-slave-9jll krb5kdc[23872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:55 dist-test-slave-9jll krb5kdc[23872](info): set up 2 sockets
Sep 27 18:43:55 dist-test-slave-9jll krb5kdc[23872](info): commencing operation
krb5kdc: starting...
W20260927 18:43:56.682534 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.007s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:56 dist-test-slave-9jll krb5kdc[23872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:43:57.006697 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:57.414477 23885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:57.414922 23885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:57.415067 23885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:57.423216 23885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:57.423368 23885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:57.423431 23885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:57.423523 23885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:57.423589 23885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:57.435035 23885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41139
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:57.436869 23885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:57.438620 23885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:57.453248 23891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:57.453190 23893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:57.453357 23885 server_base.cc:1034] running on GCE node
W20260927 18:43:57.453246 23890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:57.454754 23885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:57.456598 23885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:57.457793 23885 hybrid_clock.cc:648] HybridClock initialized: now 1790534637457730 us; error 47 us; skew 500 ppm
I20260927 18:43:57.458287 23885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:57 dist-test-slave-9jll krb5kdc[23872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:57.462347 23885 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:43:57.464320 23885 webserver.cc:460] Webserver started at http://127.0.0.1:37527/ using document root <none> and password file <none>
I20260927 18:43:57.465235 23885 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:57.465371 23885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:57.465782 23885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:57.469514 23885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "c32010a68d334c5c9ad90d8bb10626c8"
format_stamp: "Formatted at 2026-09-27 18:43:57 on dist-test-slave-9jll"
server_key: "fddba66e4c7f4e3ff5ebe199fd31be53"
server_key_iv: "2054cb4d2c56abec7a48cf4ea337fca7"
server_key_version: "encryptionkey@0"
I20260927 18:43:57.470505 23885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "c32010a68d334c5c9ad90d8bb10626c8"
format_stamp: "Formatted at 2026-09-27 18:43:57 on dist-test-slave-9jll"
server_key: "fddba66e4c7f4e3ff5ebe199fd31be53"
server_key_iv: "2054cb4d2c56abec7a48cf4ea337fca7"
server_key_version: "encryptionkey@0"
I20260927 18:43:57.477080 23885 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:43:57.481562 23900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:57.483330 23885 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:43:57.483534 23885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "c32010a68d334c5c9ad90d8bb10626c8"
format_stamp: "Formatted at 2026-09-27 18:43:57 on dist-test-slave-9jll"
server_key: "fddba66e4c7f4e3ff5ebe199fd31be53"
server_key_iv: "2054cb4d2c56abec7a48cf4ea337fca7"
server_key_version: "encryptionkey@0"
I20260927 18:43:57.483829 23885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:57.508920 23885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:57.514496 23885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:57.514823 23885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:57.535193 23885 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41139
I20260927 18:43:57.535207 23951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41139 every 8 connection(s)
I20260927 18:43:57.537052 23885 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:57.542253 23952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:57.542590 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 23885
I20260927 18:43:57.542793 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:57.543467 2429 external_mini_cluster.cc:1428] Setting key d7f18c4466556415dfc1cbb3d71b9479
I20260927 18:43:57.559689 23952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8: Bootstrap starting.
Sep 27 18:43:57 dist-test-slave-9jll krb5kdc[23872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 18:43:57.564460 23952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:57.566185 23952 log.cc:824] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:57.570429 23952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8: No bootstrap required, opened a new log
I20260927 18:43:57.571652 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:57.545771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:57409 (local address 127.0.0.1:41139)
0927 18:43:57.546756 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:57.546774 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:57.546831 (+ 57us) server_negotiation.cc:407] Connection header received
0927 18:43:57.548906 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:57.548951 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:57.549710 (+ 759us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:57.550311 (+ 601us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:57.551973 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:57.554115 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:57.555696 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:57.556885 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:57.557086 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:57.562141 (+ 5055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:57.562154 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:57.562177 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:57.562241 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:57.566523 (+ 4282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:57.567540 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:57.567554 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:57.567565 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:57.567816 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:57.568449 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:57.568452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:57.568454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:57.569129 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:57.569400 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:57.570031 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 18:43:57.570410 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":420,"threads_started":1}
I20260927 18:43:57.576654 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:43:57.579843 23952 raft_consensus.cc:348] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32010a68d334c5c9ad90d8bb10626c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41139 } }
I20260927 18:43:57.580339 23952 raft_consensus.cc:374] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:57.580463 23952 raft_consensus.cc:729] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32010a68d334c5c9ad90d8bb10626c8, State: Initialized, Role: FOLLOWER
I20260927 18:43:57.581156 23952 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32010a68d334c5c9ad90d8bb10626c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41139 } }
I20260927 18:43:57.581432 23952 raft_consensus.cc:388] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:57.581559 23952 raft_consensus.cc:482] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:57.581753 23952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:57.585330 23952 raft_consensus.cc:504] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32010a68d334c5c9ad90d8bb10626c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41139 } }
I20260927 18:43:57.586026 23952 leader_election.cc:302] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c32010a68d334c5c9ad90d8bb10626c8; no voters:
I20260927 18:43:57.586858 23952 leader_election.cc:288] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:57.586951 23962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:57.588376 23962 raft_consensus.cc:686] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [term 1 LEADER]: Becoming Leader. State: Replica: c32010a68d334c5c9ad90d8bb10626c8, State: Running, Role: LEADER
I20260927 18:43:57.589183 23962 consensus_queue.cc:237] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32010a68d334c5c9ad90d8bb10626c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41139 } }
I20260927 18:43:57.589869 23952 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:57.594399 23963 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c32010a68d334c5c9ad90d8bb10626c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32010a68d334c5c9ad90d8bb10626c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41139 } } }
I20260927 18:43:57.594300 23964 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c32010a68d334c5c9ad90d8bb10626c8. Latest consensus state: current_term: 1 leader_uuid: "c32010a68d334c5c9ad90d8bb10626c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32010a68d334c5c9ad90d8bb10626c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41139 } } }
I20260927 18:43:57.594722 23964 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:57.594722 23963 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8 [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:57.595685 23970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:57.602234 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:57.581883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:50093 (local address 127.0.0.1:41139)
0927 18:43:57.582109 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:57.582116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:57.582332 (+ 216us) server_negotiation.cc:407] Connection header received
0927 18:43:57.582518 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:57.582522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:57.582663 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:57.582766 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:57.584835 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:57.585841 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:57.587718 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:57.588426 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:57.588568 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:57.592561 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:57.592568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:57.592575 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:43:57.592627 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:57.597091 (+ 4464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:57.598530 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:57.598537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:57.598541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:57.598707 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:43:57.599387 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:43:57.599393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:43:57.599396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:43:57.599901 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:57.601318 (+ 1417us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:57.601519 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 18:43:57.601782 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 18:43:57.613089 23970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:57.624265 23970 catalog_manager.cc:1347] Generated new cluster ID: 484f813155e2494bad880744fff847f1
I20260927 18:43:57.624413 23970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:57.647524 23970 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:57.649077 23970 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:57.659714 23970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c32010a68d334c5c9ad90d8bb10626c8: Generated new TSK 0
I20260927 18:43:57.660827 23970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:57.671193 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23885
I20260927 18:43:57.672468 23947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:43:57.777393 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 23885
W20260927 18:43:57.795399 23870 connection.cc:535] client connection to 127.0.0.1:41139 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41139): Connection reset by peer (error 104)
2026-09-27T18:43:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2152 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:57 dist-test-slave-9jll krb5kdc[23984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:57 dist-test-slave-9jll krb5kdc[23984](info): set up 2 sockets
Sep 27 18:43:57 dist-test-slave-9jll krb5kdc[23984](info): commencing operation
krb5kdc: starting...
W20260927 18:43:58.838376 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:43:59 dist-test-slave-9jll krb5kdc[23984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:43:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:43:59.154081 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:43:59.544986 23997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:43:59.545399 23997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:43:59.545567 23997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:43:59.554246 23997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:43:59.554409 23997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:43:59.554464 23997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:43:59.554585 23997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:43:59.554637 23997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:43:59.565909 23997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41519
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:43:59.567703 23997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:43:59.569324 23997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:43:59.583015 24002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:59.583040 24003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:43:59.582983 24005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:43:59.587396 23997 server_base.cc:1034] running on GCE node
I20260927 18:43:59.588434 23997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:43:59.590334 23997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:43:59.591559 23997 hybrid_clock.cc:648] HybridClock initialized: now 1790534639591509 us; error 47 us; skew 500 ppm
I20260927 18:43:59.591960 23997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:43:59 dist-test-slave-9jll krb5kdc[23984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:43:59.595782 23997 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:43:59.597613 23997 webserver.cc:460] Webserver started at http://127.0.0.1:34869/ using document root <none> and password file <none>
I20260927 18:43:59.598500 23997 fs_manager.cc:359] Metadata directory not provided
I20260927 18:43:59.598608 23997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:43:59.598923 23997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:43:59.602627 23997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "1e3d8882a7d44043ad7d25b3a0dccc1d"
format_stamp: "Formatted at 2026-09-27 18:43:59 on dist-test-slave-9jll"
server_key: "9247dea2208cda55868a25beb6ca73d8"
server_key_iv: "effb35143a2ca439b312180773d05628"
server_key_version: "encryptionkey@0"
I20260927 18:43:59.603457 23997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "1e3d8882a7d44043ad7d25b3a0dccc1d"
format_stamp: "Formatted at 2026-09-27 18:43:59 on dist-test-slave-9jll"
server_key: "9247dea2208cda55868a25beb6ca73d8"
server_key_iv: "effb35143a2ca439b312180773d05628"
server_key_version: "encryptionkey@0"
I20260927 18:43:59.609895 23997 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 18:43:59.614063 24012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:43:59.615965 23997 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:43:59.616171 23997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "1e3d8882a7d44043ad7d25b3a0dccc1d"
format_stamp: "Formatted at 2026-09-27 18:43:59 on dist-test-slave-9jll"
server_key: "9247dea2208cda55868a25beb6ca73d8"
server_key_iv: "effb35143a2ca439b312180773d05628"
server_key_version: "encryptionkey@0"
I20260927 18:43:59.616398 23997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:43:59.634502 23997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:43:59.639580 23997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:43:59.639914 23997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:43:59.661381 23997 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41519
I20260927 18:43:59.661410 24063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41519 every 8 connection(s)
I20260927 18:43:59.663175 23997 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:43:59.667484 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 23997
I20260927 18:43:59.667717 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:43:59.668177 24064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:43:59.668362 2429 external_mini_cluster.cc:1428] Setting key b86df4880aa6f07faca00f949ce059f2
I20260927 18:43:59.682634 24064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d: Bootstrap starting.
I20260927 18:43:59.682986 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:59.670532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:38291 (local address 127.0.0.1:41519)
0927 18:43:59.671391 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:59.671404 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:59.671451 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:43:59.673600 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:59.673627 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:59.674188 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:59.674633 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:59.675243 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:59.676655 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:59.678110 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:59.678828 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:59.679016 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:59.680349 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:59.680360 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:59.680377 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:59.680998 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:59.681087 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:59.681114 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:59.681270 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:59.681555 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 18:43:59.681787 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":383,"threads_started":1}
I20260927 18:43:59.686445 24064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d: Neither blocks nor log segments found. Creating new log.
I20260927 18:43:59.687238 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:43:59.687707 24064 log.cc:824] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d: Log is configured to *not* fsync() on all Append() calls
I20260927 18:43:59.690867 24064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d: No bootstrap required, opened a new log
I20260927 18:43:59.699986 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:43:59.692519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:53857 (local address 127.0.0.1:41519)
0927 18:43:59.692744 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 18:43:59.692751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:43:59.693118 (+ 367us) server_negotiation.cc:407] Connection header received
0927 18:43:59.693349 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:43:59.693354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:43:59.693432 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:43:59.693530 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:43:59.694264 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:59.695201 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:59.697121 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:43:59.697618 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:43:59.697756 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:43:59.699122 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:43:59.699127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:43:59.699131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:43:59.699156 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:43:59.699206 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:43:59.699223 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:43:59.699313 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:43:59.699511 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 18:43:59.699568 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:43:59.701203 24064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3d8882a7d44043ad7d25b3a0dccc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41519 } }
I20260927 18:43:59.701777 24064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:43:59.701882 24064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3d8882a7d44043ad7d25b3a0dccc1d, State: Initialized, Role: FOLLOWER
I20260927 18:43:59.702490 24064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3d8882a7d44043ad7d25b3a0dccc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41519 } }
I20260927 18:43:59.702800 24064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:43:59.702909 24064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:43:59.703049 24064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:43:59.705382 24064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3d8882a7d44043ad7d25b3a0dccc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41519 } }
I20260927 18:43:59.705982 24064 leader_election.cc:302] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e3d8882a7d44043ad7d25b3a0dccc1d; no voters:
I20260927 18:43:59.706660 24064 leader_election.cc:288] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:43:59.706786 24074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:43:59.708287 24074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [term 1 LEADER]: Becoming Leader. State: Replica: 1e3d8882a7d44043ad7d25b3a0dccc1d, State: Running, Role: LEADER
I20260927 18:43:59.708953 24074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3d8882a7d44043ad7d25b3a0dccc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41519 } }
I20260927 18:43:59.709172 24064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:43:59.712009 24076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e3d8882a7d44043ad7d25b3a0dccc1d. Latest consensus state: current_term: 1 leader_uuid: "1e3d8882a7d44043ad7d25b3a0dccc1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3d8882a7d44043ad7d25b3a0dccc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41519 } } }
I20260927 18:43:59.712448 24076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:59.712811 24075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e3d8882a7d44043ad7d25b3a0dccc1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e3d8882a7d44043ad7d25b3a0dccc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41519 } } }
I20260927 18:43:59.713117 24075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d [sys.catalog]: This master's current role is: LEADER
I20260927 18:43:59.715551 24080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:43:59.720741 24080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:43:59.731331 24080 catalog_manager.cc:1347] Generated new cluster ID: 0cb70609854248d3a3f292139498743e
I20260927 18:43:59.731462 24080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:43:59.757503 24080 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:43:59.759126 24080 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:43:59.772568 24080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e3d8882a7d44043ad7d25b3a0dccc1d: Generated new TSK 0
I20260927 18:43:59.773739 24080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:43:59.786890 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23997
W20260927 18:43:59.788745 24059 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.62:38291
W20260927 18:43:59.789618 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 23997: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:43:59.789815 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 23997
W20260927 18:43:59.807742 23982 connection.cc:535] client connection to 127.0.0.1:41519 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41519): Connection reset by peer (error 104)
2026-09-27T18:43:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2012 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:43:59 dist-test-slave-9jll krb5kdc[24095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:43:59 dist-test-slave-9jll krb5kdc[24095](info): set up 2 sockets
Sep 27 18:43:59 dist-test-slave-9jll krb5kdc[24095](info): commencing operation
krb5kdc: starting...
W20260927 18:44:00.854514 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:44:01 dist-test-slave-9jll krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 18:44:01.173444 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:01.563421 24108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:01.563819 24108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:01.563956 24108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:01.572154 24108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:01.572320 24108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:01.572402 24108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:01.572494 24108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:01.572558 24108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:01.583779 24108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:01.585523 24108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:01.587273 24108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:01.600944 24113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:01.600968 24116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:01.600864 24114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:01.601809 24108 server_base.cc:1034] running on GCE node
I20260927 18:44:01.602694 24108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:01.604355 24108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:01.605582 24108 hybrid_clock.cc:648] HybridClock initialized: now 1790534641605525 us; error 50 us; skew 500 ppm
I20260927 18:44:01.606010 24108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:01 dist-test-slave-9jll krb5kdc[24095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:01.618165 24108 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 18:44:01.620342 24108 webserver.cc:460] Webserver started at http://127.0.0.1:35025/ using document root <none> and password file <none>
I20260927 18:44:01.621222 24108 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:01.621359 24108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:01.621765 24108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:01.625425 24108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "6d84097948f74ecba0321baae556adfa"
format_stamp: "Formatted at 2026-09-27 18:44:01 on dist-test-slave-9jll"
server_key: "9dd3cd4b8687d215ab6735a8737b0177"
server_key_iv: "f3d81b3269ba6a3c49221b8372fd3c78"
server_key_version: "encryptionkey@0"
I20260927 18:44:01.626346 24108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "6d84097948f74ecba0321baae556adfa"
format_stamp: "Formatted at 2026-09-27 18:44:01 on dist-test-slave-9jll"
server_key: "9dd3cd4b8687d215ab6735a8737b0177"
server_key_iv: "f3d81b3269ba6a3c49221b8372fd3c78"
server_key_version: "encryptionkey@0"
I20260927 18:44:01.632560 24108 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 18:44:01.636940 24123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:01.638706 24108 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:44:01.638902 24108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "6d84097948f74ecba0321baae556adfa"
format_stamp: "Formatted at 2026-09-27 18:44:01 on dist-test-slave-9jll"
server_key: "9dd3cd4b8687d215ab6735a8737b0177"
server_key_iv: "f3d81b3269ba6a3c49221b8372fd3c78"
server_key_version: "encryptionkey@0"
I20260927 18:44:01.639139 24108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:01.646373 24108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:01.651470 24108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:01.651826 24108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:01.672770 24108 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35955
I20260927 18:44:01.672796 24174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35955 every 8 connection(s)
I20260927 18:44:01.674517 24108 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:01.676008 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 24108
I20260927 18:44:01.676250 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:44:01.676904 2429 external_mini_cluster.cc:1428] Setting key b7f9e761acadf83f814d1f8259512b5d
I20260927 18:44:01.680063 24175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:01.685885 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:01.679210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:60245 (local address 127.0.0.1:35955)
0927 18:44:01.680093 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:01.680106 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:01.680166 (+ 60us) server_negotiation.cc:407] Connection header received
0927 18:44:01.682045 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:01.682071 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:01.682443 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:01.682819 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:01.683990 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:01.683998 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:01.684011 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:44:01.684029 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:01.684113 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:44:01.684136 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:01.684324 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:01.684435 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 18:44:01.684748 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":374,"threads_started":1}
I20260927 18:44:01.690085 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:44:01.694563 24175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa: Bootstrap starting.
I20260927 18:44:01.699100 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:01.696192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:46183 (local address 127.0.0.1:35955)
0927 18:44:01.696360 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:01.696365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:01.696753 (+ 388us) server_negotiation.cc:407] Connection header received
0927 18:44:01.697082 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:01.697088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:01.697158 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:01.697254 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:01.698362 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:01.698367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:01.698371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:44:01.698388 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:01.698438 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:44:01.698455 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:01.698546 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:01.698715 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 18:44:01.698778 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 18:44:01.699508 24175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:01.700863 24175 log.cc:824] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:01.704303 24175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa: No bootstrap required, opened a new log
I20260927 18:44:01.713529 24175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d84097948f74ecba0321baae556adfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35955 } }
I20260927 18:44:01.714274 24175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:01.714442 24175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d84097948f74ecba0321baae556adfa, State: Initialized, Role: FOLLOWER
I20260927 18:44:01.715286 24175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d84097948f74ecba0321baae556adfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35955 } }
I20260927 18:44:01.715544 24175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:01.715719 24175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:01.715878 24175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:01.718477 24175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d84097948f74ecba0321baae556adfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35955 } }
I20260927 18:44:01.719084 24175 leader_election.cc:302] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d84097948f74ecba0321baae556adfa; no voters:
I20260927 18:44:01.719782 24175 leader_election.cc:288] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:01.719944 24185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:01.721259 24185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [term 1 LEADER]: Becoming Leader. State: Replica: 6d84097948f74ecba0321baae556adfa, State: Running, Role: LEADER
I20260927 18:44:01.721989 24185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d84097948f74ecba0321baae556adfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35955 } }
I20260927 18:44:01.722529 24175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:01.725385 24186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d84097948f74ecba0321baae556adfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d84097948f74ecba0321baae556adfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35955 } } }
I20260927 18:44:01.725282 24187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d84097948f74ecba0321baae556adfa. Latest consensus state: current_term: 1 leader_uuid: "6d84097948f74ecba0321baae556adfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d84097948f74ecba0321baae556adfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35955 } } }
I20260927 18:44:01.725963 24186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:01.725977 24187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:01.728449 24194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:01.733882 24194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:01.744859 24194 catalog_manager.cc:1347] Generated new cluster ID: 05d3bbb71f6c462e8d3463739826c4fd
I20260927 18:44:01.745024 24194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:01.762220 24194 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:01.762809 24194 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:01.776872 24194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d84097948f74ecba0321baae556adfa: Generated new TSK 0
I20260927 18:44:01.778416 24194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:44:01.792371 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24108
W20260927 18:44:01.793803 24170 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.62:60245
W20260927 18:44:01.794864 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24108: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:01.795040 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 24108
2026-09-27T18:44:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2005 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:01 dist-test-slave-9jll krb5kdc[24206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:01 dist-test-slave-9jll krb5kdc[24206](info): set up 2 sockets
Sep 27 18:44:01 dist-test-slave-9jll krb5kdc[24206](info): commencing operation
krb5kdc: starting...
W20260927 18:44:02.846421 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:44:03 dist-test-slave-9jll krb5kdc[24206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:44:03.174227 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33197
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:33197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:03.576129 24219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:03.576483 24219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:03.576623 24219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:03.584599 24219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:03.584745 24219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:03.584797 24219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:03.584883 24219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:03.584947 24219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:03.596092 24219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:33197
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:03.597886 24219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:03.599524 24219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:03.612828 24224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:03.612896 24227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:03.612931 24225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:03.613638 24219 server_base.cc:1034] running on GCE node
I20260927 18:44:03.614641 24219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:03.616490 24219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:03.617774 24219 hybrid_clock.cc:648] HybridClock initialized: now 1790534643617709 us; error 57 us; skew 500 ppm
I20260927 18:44:03.618218 24219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:03 dist-test-slave-9jll krb5kdc[24206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534643, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:03.622059 24219 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:44:03.623880 24219 webserver.cc:460] Webserver started at http://127.2.95.126:38831/ using document root <none> and password file <none>
I20260927 18:44:03.624835 24219 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:03.624957 24219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:03.625308 24219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:03.629037 24219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "fefd3e871cb840af95e070102ca1fb97"
format_stamp: "Formatted at 2026-09-27 18:44:03 on dist-test-slave-9jll"
server_key: "19714c4b9a0802e357f00d9a5d5af1fc"
server_key_iv: "9330679a2eec0dc8f2dfad69a6126815"
server_key_version: "encryptionkey@0"
I20260927 18:44:03.630046 24219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "fefd3e871cb840af95e070102ca1fb97"
format_stamp: "Formatted at 2026-09-27 18:44:03 on dist-test-slave-9jll"
server_key: "19714c4b9a0802e357f00d9a5d5af1fc"
server_key_iv: "9330679a2eec0dc8f2dfad69a6126815"
server_key_version: "encryptionkey@0"
I20260927 18:44:03.636351 24219 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:44:03.640715 24234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:03.642460 24219 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 18:44:03.642659 24219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "fefd3e871cb840af95e070102ca1fb97"
format_stamp: "Formatted at 2026-09-27 18:44:03 on dist-test-slave-9jll"
server_key: "19714c4b9a0802e357f00d9a5d5af1fc"
server_key_iv: "9330679a2eec0dc8f2dfad69a6126815"
server_key_version: "encryptionkey@0"
I20260927 18:44:03.642892 24219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:03.669519 24219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:03.674791 24219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:03.675098 24219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:03.696130 24219 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:33197
I20260927 18:44:03.696172 24285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:33197 every 8 connection(s)
I20260927 18:44:03.697916 24219 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:03.702796 24286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:03.708223 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 24219
I20260927 18:44:03.708437 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:44:03.709064 2429 external_mini_cluster.cc:1428] Setting key 335b6661b02228c97dda27b07770dbd6
I20260927 18:44:03.716322 24286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97: Bootstrap starting.
I20260927 18:44:03.720988 24286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:03.722507 24286 log.cc:824] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97: Log is configured to *not* fsync() on all Append() calls
Sep 27 18:44:03 dist-test-slave-9jll krb5kdc[24206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:44:03.726436 24286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97: No bootstrap required, opened a new log
I20260927 18:44:03.736086 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:03.711303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60647 (local address 127.2.95.126:33197)
0927 18:44:03.712226 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:03.712242 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:03.712292 (+ 50us) server_negotiation.cc:407] Connection header received
0927 18:44:03.714460 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:03.714492 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:03.715028 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:03.715429 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:03.717470 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:03.719082 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:03.720931 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:03.721846 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:03.721992 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:03.727398 (+ 5406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:03.727406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:03.727426 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:03.727474 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:03.730801 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:03.731893 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:03.731901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:03.731912 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:03.732136 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:03.732623 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:03.732627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:03.732629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:03.733307 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:03.733611 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:03.734312 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 18:44:03.734739 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":418,"threads_started":1}
I20260927 18:44:03.736663 24286 raft_consensus.cc:348] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fefd3e871cb840af95e070102ca1fb97" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33197 } }
I20260927 18:44:03.737098 24286 raft_consensus.cc:374] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:03.737224 24286 raft_consensus.cc:729] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fefd3e871cb840af95e070102ca1fb97, State: Initialized, Role: FOLLOWER
I20260927 18:44:03.737939 24286 consensus_queue.cc:260] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fefd3e871cb840af95e070102ca1fb97" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33197 } }
I20260927 18:44:03.738202 24286 raft_consensus.cc:388] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:03.738339 24286 raft_consensus.cc:482] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:03.738523 24286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:03.740679 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:44:03.741652 24286 raft_consensus.cc:504] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fefd3e871cb840af95e070102ca1fb97" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33197 } }
I20260927 18:44:03.742429 24286 leader_election.cc:302] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fefd3e871cb840af95e070102ca1fb97; no voters:
I20260927 18:44:03.743225 24286 leader_election.cc:288] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:03.743352 24291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:03.744637 24291 raft_consensus.cc:686] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [term 1 LEADER]: Becoming Leader. State: Replica: fefd3e871cb840af95e070102ca1fb97, State: Running, Role: LEADER
I20260927 18:44:03.745395 24291 consensus_queue.cc:237] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fefd3e871cb840af95e070102ca1fb97" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33197 } }
I20260927 18:44:03.746114 24286 sys_catalog.cc:564] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:03.751403 24292 sys_catalog.cc:455] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fefd3e871cb840af95e070102ca1fb97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fefd3e871cb840af95e070102ca1fb97" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33197 } } }
I20260927 18:44:03.751905 24292 sys_catalog.cc:458] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:03.752468 24293 sys_catalog.cc:455] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fefd3e871cb840af95e070102ca1fb97. Latest consensus state: current_term: 1 leader_uuid: "fefd3e871cb840af95e070102ca1fb97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fefd3e871cb840af95e070102ca1fb97" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33197 } } }
I20260927 18:44:03.752751 24293 sys_catalog.cc:458] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:03.754777 24304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:03.761097 24304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:03.772454 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:03.747544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50559 (local address 127.2.95.126:33197)
0927 18:44:03.747875 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:03.747883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:03.749765 (+ 1882us) server_negotiation.cc:407] Connection header received
0927 18:44:03.751069 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:03.751209 (+ 140us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:03.751331 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:03.751441 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:03.754128 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:03.755361 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:03.757433 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:03.758243 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:03.758361 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:03.762628 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:03.762637 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:03.762644 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:03.762699 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:03.767287 (+ 4588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:03.768730 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:03.768737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:03.768742 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:03.768930 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:03.769850 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:03.769858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:03.769862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:03.770493 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:03.770845 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:03.771697 (+ 852us) server_negotiation.cc:299] Negotiation successful
0927 18:44:03.771980 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260927 18:44:03.784721 24304 catalog_manager.cc:1347] Generated new cluster ID: 8863047272804aa3aaacb8499c74a246
I20260927 18:44:03.784888 24304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:03.814666 24304 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:03.816823 24304 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:03.825392 24304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fefd3e871cb840af95e070102ca1fb97: Generated new TSK 0
I20260927 18:44:03.826745 24304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:44:03.841938 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24219
I20260927 18:44:03.844135 24281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:03.951143 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 24219
W20260927 18:44:03.968994 24204 connection.cc:535] client connection to 127.2.95.126:33197 recv error: Network error: failed to read from TLS socket (remote: 127.2.95.126:33197): Connection reset by peer (error 104)
2026-09-27T18:44:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2156 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:04 dist-test-slave-9jll krb5kdc[24318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:04 dist-test-slave-9jll krb5kdc[24318](info): set up 2 sockets
Sep 27 18:44:04 dist-test-slave-9jll krb5kdc[24318](info): commencing operation
krb5kdc: starting...
W20260927 18:44:05.022429 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.008s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:44:05 dist-test-slave-9jll krb5kdc[24318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:44:05.334046 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:36883
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:36883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:05.731539 24331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:05.731985 24331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:05.732148 24331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:05.740419 24331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:05.740624 24331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:05.740731 24331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:05.740823 24331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:05.740885 24331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:05.753046 24331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:36883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:36883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:05.754900 24331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:05.756611 24331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:05.770428 24336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:05.770375 24337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:05.770403 24339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:05.771448 24331 server_base.cc:1034] running on GCE node
I20260927 18:44:05.772461 24331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:05.774488 24331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:05.775774 24331 hybrid_clock.cc:648] HybridClock initialized: now 1790534645775707 us; error 62 us; skew 500 ppm
I20260927 18:44:05.776240 24331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:05 dist-test-slave-9jll krb5kdc[24318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534645, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:05.780228 24331 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:44:05.782267 24331 webserver.cc:460] Webserver started at http://127.2.95.126:40311/ using document root <none> and password file <none>
I20260927 18:44:05.783146 24331 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:05.783281 24331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:05.783627 24331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:05.787425 24331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "3a17d530285446b8803c9b5e78847702"
format_stamp: "Formatted at 2026-09-27 18:44:05 on dist-test-slave-9jll"
server_key: "00c2b57fdfa7e5a091396fe7adbfd605"
server_key_iv: "8c767855d7ca7e31df9be7d8ee63ad3e"
server_key_version: "encryptionkey@0"
I20260927 18:44:05.788343 24331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "3a17d530285446b8803c9b5e78847702"
format_stamp: "Formatted at 2026-09-27 18:44:05 on dist-test-slave-9jll"
server_key: "00c2b57fdfa7e5a091396fe7adbfd605"
server_key_iv: "8c767855d7ca7e31df9be7d8ee63ad3e"
server_key_version: "encryptionkey@0"
I20260927 18:44:05.795069 24331 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 18:44:05.800156 24346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:05.802162 24331 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 18:44:05.802397 24331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "3a17d530285446b8803c9b5e78847702"
format_stamp: "Formatted at 2026-09-27 18:44:05 on dist-test-slave-9jll"
server_key: "00c2b57fdfa7e5a091396fe7adbfd605"
server_key_iv: "8c767855d7ca7e31df9be7d8ee63ad3e"
server_key_version: "encryptionkey@0"
I20260927 18:44:05.802668 24331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:05.838235 24331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:05.843760 24331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:05.844129 24331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:05.867118 24331 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:36883
I20260927 18:44:05.867149 24397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:36883 every 8 connection(s)
I20260927 18:44:05.868924 24331 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:05.874214 24398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:05.878348 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 24331
I20260927 18:44:05.878562 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:44:05.879189 2429 external_mini_cluster.cc:1428] Setting key 2ae89f55f58dcf8abb1345cd8795fc2f
I20260927 18:44:05.889446 24398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702: Bootstrap starting.
I20260927 18:44:05.894035 24398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:05.895390 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:05.881647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55473 (local address 127.2.95.126:36883)
0927 18:44:05.882736 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:05.882751 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:05.882804 (+ 53us) server_negotiation.cc:407] Connection header received
0927 18:44:05.884709 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:05.884741 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:05.885285 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:05.885734 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:05.886425 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:05.887880 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:05.890038 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:05.890879 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:05.891042 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:05.892543 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:05.892559 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:05.892577 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:44:05.893248 (+ 671us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:05.893338 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:44:05.893373 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:05.893500 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:05.893889 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 18:44:05.894167 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":477,"threads_started":1}
I20260927 18:44:05.895370 24398 log.cc:824] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:05.899493 24398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702: No bootstrap required, opened a new log
I20260927 18:44:05.900550 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:44:05.910876 24398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a17d530285446b8803c9b5e78847702" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 36883 } }
I20260927 18:44:05.911537 24398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:05.911685 24398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a17d530285446b8803c9b5e78847702, State: Initialized, Role: FOLLOWER
I20260927 18:44:05.912426 24398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a17d530285446b8803c9b5e78847702" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 36883 } }
I20260927 18:44:05.912703 24398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:05.912856 24398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:05.913079 24398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:05.913904 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:05.906428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54693 (local address 127.2.95.126:36883)
0927 18:44:05.906643 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:05.906651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:05.907196 (+ 545us) server_negotiation.cc:407] Connection header received
0927 18:44:05.907466 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:05.907472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:05.907571 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:05.907690 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:05.908226 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:05.909482 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:05.910926 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:05.911584 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:05.911720 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:05.912958 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:05.912965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:05.912969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:44:05.912994 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:05.913071 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:44:05.913098 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:05.913232 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:05.913476 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 18:44:05.913540 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 18:44:05.916293 24398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a17d530285446b8803c9b5e78847702" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 36883 } }
I20260927 18:44:05.916858 24398 leader_election.cc:302] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a17d530285446b8803c9b5e78847702; no voters:
I20260927 18:44:05.917439 24398 leader_election.cc:288] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:05.917591 24408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:05.918403 24408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [term 1 LEADER]: Becoming Leader. State: Replica: 3a17d530285446b8803c9b5e78847702, State: Running, Role: LEADER
I20260927 18:44:05.919056 24408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a17d530285446b8803c9b5e78847702" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 36883 } }
I20260927 18:44:05.919952 24398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:05.922206 24409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a17d530285446b8803c9b5e78847702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a17d530285446b8803c9b5e78847702" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 36883 } } }
I20260927 18:44:05.922549 24409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:05.924724 24410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a17d530285446b8803c9b5e78847702. Latest consensus state: current_term: 1 leader_uuid: "3a17d530285446b8803c9b5e78847702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a17d530285446b8803c9b5e78847702" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 36883 } } }
I20260927 18:44:05.925066 24410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:05.923552 24415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:05.935840 24415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:05.948482 24415 catalog_manager.cc:1347] Generated new cluster ID: 77933cd62c314beb80874be28f08a4a6
I20260927 18:44:05.948719 24415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:05.962481 24415 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:05.964120 24415 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:05.980473 24415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a17d530285446b8803c9b5e78847702: Generated new TSK 0
I20260927 18:44:05.981751 24415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:44:05.997442 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24331
W20260927 18:44:05.999260 24393 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55473
W20260927 18:44:06.000176 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24331: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:06.000331 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 24331
W20260927 18:44:06.018620 24316 connection.cc:535] client connection to 127.2.95.126:36883 recv error: Network error: failed to read from TLS socket (remote: 127.2.95.126:36883): Connection reset by peer (error 104)
2026-09-27T18:44:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:06 dist-test-slave-9jll krb5kdc[24429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:06 dist-test-slave-9jll krb5kdc[24429](info): set up 2 sockets
Sep 27 18:44:06 dist-test-slave-9jll krb5kdc[24429](info): commencing operation
krb5kdc: starting...
W20260927 18:44:07.062414 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.007s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:44:07 dist-test-slave-9jll krb5kdc[24429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:44:07.390444 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:40847
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:40847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:07.795070 24442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:07.795464 24442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:07.795624 24442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:07.803856 24442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:07.804011 24442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:07.804066 24442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:07.804162 24442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:07.804229 24442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:07.816025 24442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:40847
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:40847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:07.817869 24442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:07.819701 24442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:07.833614 24447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:07.833580 24448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:07.833776 24450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:07.835423 24442 server_base.cc:1034] running on GCE node
I20260927 18:44:07.836517 24442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:07.838557 24442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:07.839826 24442 hybrid_clock.cc:648] HybridClock initialized: now 1790534647839747 us; error 66 us; skew 500 ppm
I20260927 18:44:07.840283 24442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:07 dist-test-slave-9jll krb5kdc[24429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534647, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:07.849989 24442 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:44:07.852208 24442 webserver.cc:460] Webserver started at http://127.2.95.126:35593/ using document root <none> and password file <none>
I20260927 18:44:07.853147 24442 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:07.853295 24442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:07.853698 24442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:07.857475 24442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "733d34a588df41d3b58d232ff76913e3"
format_stamp: "Formatted at 2026-09-27 18:44:07 on dist-test-slave-9jll"
server_key: "d2b308f2b33b55051ad7e0d62eb9a468"
server_key_iv: "a47d20c04632712c4572186d93115079"
server_key_version: "encryptionkey@0"
I20260927 18:44:07.858498 24442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "733d34a588df41d3b58d232ff76913e3"
format_stamp: "Formatted at 2026-09-27 18:44:07 on dist-test-slave-9jll"
server_key: "d2b308f2b33b55051ad7e0d62eb9a468"
server_key_iv: "a47d20c04632712c4572186d93115079"
server_key_version: "encryptionkey@0"
I20260927 18:44:07.865568 24442 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 18:44:07.870256 24457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:07.872123 24442 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 18:44:07.872344 24442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "733d34a588df41d3b58d232ff76913e3"
format_stamp: "Formatted at 2026-09-27 18:44:07 on dist-test-slave-9jll"
server_key: "d2b308f2b33b55051ad7e0d62eb9a468"
server_key_iv: "a47d20c04632712c4572186d93115079"
server_key_version: "encryptionkey@0"
I20260927 18:44:07.872601 24442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:07.879741 24442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:07.885280 24442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:07.885651 24442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:07.909005 24442 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:40847
I20260927 18:44:07.909045 24508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:40847 every 8 connection(s)
I20260927 18:44:07.910954 24442 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:07.915719 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 24442
I20260927 18:44:07.915943 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:44:07.916460 2429 external_mini_cluster.cc:1428] Setting key f89922d899117f2f30fdcafc04938e42
I20260927 18:44:07.916697 24509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:07.927367 24512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:07.918899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48231 (local address 127.2.95.126:40847)
0927 18:44:07.919995 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:07.920013 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:07.920161 (+ 148us) server_negotiation.cc:407] Connection header received
0927 18:44:07.922778 (+ 2617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:07.922813 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:07.923378 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:07.923854 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:07.925253 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:07.925265 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:07.925283 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:44:07.925308 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:07.925405 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:44:07.925437 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:07.925767 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:07.925908 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 18:44:07.926178 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":471,"threads_started":1}
I20260927 18:44:07.931924 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:44:07.932853 24509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3: Bootstrap starting.
I20260927 18:44:07.936991 24509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:07.938788 24509 log.cc:824] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:07.940697 24512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:07.937333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58319 (local address 127.2.95.126:40847)
0927 18:44:07.937576 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:07.937584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:07.937965 (+ 381us) server_negotiation.cc:407] Connection header received
0927 18:44:07.938149 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:07.938155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:07.938237 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:07.938358 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:07.939806 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:07.939811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:07.939815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:44:07.939838 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:07.939886 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:44:07.939904 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:07.940011 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:07.940221 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 18:44:07.940299 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:44:07.943077 24509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3: No bootstrap required, opened a new log
I20260927 18:44:07.952172 24509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "733d34a588df41d3b58d232ff76913e3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40847 } }
I20260927 18:44:07.952783 24509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:07.952890 24509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733d34a588df41d3b58d232ff76913e3, State: Initialized, Role: FOLLOWER
I20260927 18:44:07.953589 24509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "733d34a588df41d3b58d232ff76913e3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40847 } }
I20260927 18:44:07.953917 24509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:07.954062 24509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:07.954236 24509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:07.956805 24509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "733d34a588df41d3b58d232ff76913e3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40847 } }
I20260927 18:44:07.957434 24509 leader_election.cc:302] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 733d34a588df41d3b58d232ff76913e3; no voters:
I20260927 18:44:07.958264 24509 leader_election.cc:288] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:07.958436 24519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:07.959676 24519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [term 1 LEADER]: Becoming Leader. State: Replica: 733d34a588df41d3b58d232ff76913e3, State: Running, Role: LEADER
I20260927 18:44:07.960297 24519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "733d34a588df41d3b58d232ff76913e3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40847 } }
I20260927 18:44:07.960834 24509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:07.963411 24521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 733d34a588df41d3b58d232ff76913e3. Latest consensus state: current_term: 1 leader_uuid: "733d34a588df41d3b58d232ff76913e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "733d34a588df41d3b58d232ff76913e3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40847 } } }
I20260927 18:44:07.963874 24521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:07.964395 24520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "733d34a588df41d3b58d232ff76913e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "733d34a588df41d3b58d232ff76913e3" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40847 } } }
I20260927 18:44:07.964720 24520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:07.965101 24527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:07.971938 24527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:07.982254 24527 catalog_manager.cc:1347] Generated new cluster ID: e1be65a3ea6147788ccc4601537fd0d1
I20260927 18:44:07.982404 24527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:07.992316 24527 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:07.992848 24527 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:08.003173 24527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 733d34a588df41d3b58d232ff76913e3: Generated new TSK 0
I20260927 18:44:08.004352 24527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:44:08.013304 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24442
W20260927 18:44:08.015146 24504 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48231
W20260927 18:44:08.015923 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24442: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:08.016098 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 24442
2026-09-27T18:44:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:08 dist-test-slave-9jll krb5kdc[24540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:08 dist-test-slave-9jll krb5kdc[24540](info): set up 2 sockets
Sep 27 18:44:08 dist-test-slave-9jll krb5kdc[24540](info): commencing operation
krb5kdc: starting...
W20260927 18:44:09.078444 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.006s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:44:09 dist-test-slave-9jll krb5kdc[24540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:44:09.389747 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33025
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:33025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:09.795540 24553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:09.795969 24553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:09.796119 24553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:09.804334 24553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:09.804530 24553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:09.804627 24553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:09.804728 24553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:09.804790 24553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:09.817176 24553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:33025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:09.819016 24553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:09.820679 24553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 18:44:09.836521 24553 server_base.cc:1034] running on GCE node
W20260927 18:44:09.836406 24558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:09.836408 24559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:09.836476 24561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:09.838044 24553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:09.840036 24553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:09.841287 24553 hybrid_clock.cc:648] HybridClock initialized: now 1790534649841234 us; error 43 us; skew 500 ppm
I20260927 18:44:09.841806 24553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:09 dist-test-slave-9jll krb5kdc[24540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534649, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:09.845471 24553 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:44:09.847460 24553 webserver.cc:460] Webserver started at http://127.2.95.126:34413/ using document root <none> and password file <none>
I20260927 18:44:09.848383 24553 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:09.848510 24553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:09.848876 24553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:09.852754 24553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "d7e46497927848c8a641c1a067cec7d0"
format_stamp: "Formatted at 2026-09-27 18:44:09 on dist-test-slave-9jll"
server_key: "1a670265d31dcee41ed54c78d53b4329"
server_key_iv: "3a3457fe3bf323916fc44af51cf269e6"
server_key_version: "encryptionkey@0"
I20260927 18:44:09.853816 24553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "d7e46497927848c8a641c1a067cec7d0"
format_stamp: "Formatted at 2026-09-27 18:44:09 on dist-test-slave-9jll"
server_key: "1a670265d31dcee41ed54c78d53b4329"
server_key_iv: "3a3457fe3bf323916fc44af51cf269e6"
server_key_version: "encryptionkey@0"
I20260927 18:44:09.860678 24553 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:44:09.865211 24568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:09.867233 24553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 18:44:09.867465 24553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "d7e46497927848c8a641c1a067cec7d0"
format_stamp: "Formatted at 2026-09-27 18:44:09 on dist-test-slave-9jll"
server_key: "1a670265d31dcee41ed54c78d53b4329"
server_key_iv: "3a3457fe3bf323916fc44af51cf269e6"
server_key_version: "encryptionkey@0"
I20260927 18:44:09.867723 24553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:09.897230 24553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:09.902856 24553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:09.903211 24553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:09.924819 24553 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:33025
I20260927 18:44:09.924837 24619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:33025 every 8 connection(s)
I20260927 18:44:09.926676 24553 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:09.931785 24620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:09.934633 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 24553
I20260927 18:44:09.934855 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:44:09.935467 2429 external_mini_cluster.cc:1428] Setting key 304d284ff937e4ce34ff6652ff116903
I20260927 18:44:09.947330 24620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0: Bootstrap starting.
I20260927 18:44:09.952330 24620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0: Neither blocks nor log segments found. Creating new log.
Sep 27 18:44:09 dist-test-slave-9jll krb5kdc[24540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:44:09.953994 24620 log.cc:824] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:09.958127 24620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0: No bootstrap required, opened a new log
I20260927 18:44:09.963791 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:09.937925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38631 (local address 127.2.95.126:33025)
0927 18:44:09.939034 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:09.939049 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:09.939108 (+ 59us) server_negotiation.cc:407] Connection header received
0927 18:44:09.941005 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:09.941035 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:09.941555 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:09.942000 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:09.944202 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:09.945825 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:09.947821 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:09.948664 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:09.948825 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:09.954732 (+ 5907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:09.954748 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:09.954779 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:09.954859 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:09.958548 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:09.959569 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:09.959579 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:09.959587 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:09.959820 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:09.960303 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:09.960307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:09.960310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:09.961069 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:09.961350 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:09.961931 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 18:44:09.962447 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":467,"threads_started":1}
I20260927 18:44:09.968336 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:44:09.968039 24620 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e46497927848c8a641c1a067cec7d0" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33025 } }
I20260927 18:44:09.968537 24620 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:09.968642 24620 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7e46497927848c8a641c1a067cec7d0, State: Initialized, Role: FOLLOWER
I20260927 18:44:09.969354 24620 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e46497927848c8a641c1a067cec7d0" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33025 } }
I20260927 18:44:09.969655 24620 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:09.969847 24620 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:09.970021 24620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:09.973645 24620 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e46497927848c8a641c1a067cec7d0" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33025 } }
I20260927 18:44:09.974452 24620 leader_election.cc:302] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7e46497927848c8a641c1a067cec7d0; no voters:
I20260927 18:44:09.975302 24620 leader_election.cc:288] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:09.975365 24630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:09.976099 24630 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [term 1 LEADER]: Becoming Leader. State: Replica: d7e46497927848c8a641c1a067cec7d0, State: Running, Role: LEADER
I20260927 18:44:09.976819 24630 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e46497927848c8a641c1a067cec7d0" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33025 } }
I20260927 18:44:09.979303 24631 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7e46497927848c8a641c1a067cec7d0. Latest consensus state: current_term: 1 leader_uuid: "d7e46497927848c8a641c1a067cec7d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e46497927848c8a641c1a067cec7d0" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33025 } } }
I20260927 18:44:09.981101 24631 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:09.981734 24620 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:09.988986 24632 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7e46497927848c8a641c1a067cec7d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e46497927848c8a641c1a067cec7d0" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33025 } } }
I20260927 18:44:09.990723 24632 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:09.993218 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:09.974360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35333 (local address 127.2.95.126:33025)
0927 18:44:09.974605 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:09.974613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:09.975072 (+ 459us) server_negotiation.cc:407] Connection header received
0927 18:44:09.975307 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:09.975313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:09.975418 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:09.975538 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:09.977952 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:09.979179 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:09.980617 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:09.981368 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:09.981505 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:09.985263 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:09.985271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:09.985278 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:09.985333 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:09.989970 (+ 4637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:09.990970 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:09.990976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:09.990981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:09.991123 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:09.991520 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:09.991526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:09.991530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:09.991943 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:09.992220 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:09.992576 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 18:44:09.992838 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 18:44:09.994889 24634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:10.001909 24634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:10.016111 24634 catalog_manager.cc:1347] Generated new cluster ID: afa3ec2c9ae44faea3e3d593908f356d
I20260927 18:44:10.016290 24634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:10.044122 24634 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:10.045958 24634 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:10.055696 24634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7e46497927848c8a641c1a067cec7d0: Generated new TSK 0
I20260927 18:44:10.056880 24634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:44:10.066718 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24553
I20260927 18:44:10.068066 24615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:10.176040 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 24553
2026-09-27T18:44:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2160 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:10 dist-test-slave-9jll krb5kdc[24652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:10 dist-test-slave-9jll krb5kdc[24652](info): set up 2 sockets
Sep 27 18:44:10 dist-test-slave-9jll krb5kdc[24652](info): commencing operation
krb5kdc: starting...
W20260927 18:44:11.242590 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:44:11 dist-test-slave-9jll krb5kdc[24652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:44:11.557937 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:40239
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:40239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:11.952208 24665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:11.952589 24665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:11.952689 24665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:11.960770 24665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:11.960917 24665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:11.960968 24665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:11.961024 24665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:11.961071 24665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:11.972165 24665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:40239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:40239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:11.973878 24665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:11.975469 24665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:11.989192 24670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:11.989225 24671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:11.989148 24673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:11.989972 24665 server_base.cc:1034] running on GCE node
I20260927 18:44:11.990959 24665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:11.992727 24665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:11.993988 24665 hybrid_clock.cc:648] HybridClock initialized: now 1790534651993931 us; error 43 us; skew 500 ppm
I20260927 18:44:11.994426 24665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:11 dist-test-slave-9jll krb5kdc[24652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534651, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:11.998406 24665 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:44:12.000479 24665 webserver.cc:460] Webserver started at http://127.2.95.126:45347/ using document root <none> and password file <none>
I20260927 18:44:12.001376 24665 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:12.001552 24665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:12.002008 24665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:12.006408 24665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "d2ac4ec96b214f1b8a3ffedd8d153603"
format_stamp: "Formatted at 2026-09-27 18:44:12 on dist-test-slave-9jll"
server_key: "919da45ba49d5834e3b45b0c1f4084c2"
server_key_iv: "1394a464171abe35c7218cc20cd32229"
server_key_version: "encryptionkey@0"
I20260927 18:44:12.007471 24665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "d2ac4ec96b214f1b8a3ffedd8d153603"
format_stamp: "Formatted at 2026-09-27 18:44:12 on dist-test-slave-9jll"
server_key: "919da45ba49d5834e3b45b0c1f4084c2"
server_key_iv: "1394a464171abe35c7218cc20cd32229"
server_key_version: "encryptionkey@0"
I20260927 18:44:12.014858 24665 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 18:44:12.019719 24680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:12.021543 24665 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 18:44:12.021827 24665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "d2ac4ec96b214f1b8a3ffedd8d153603"
format_stamp: "Formatted at 2026-09-27 18:44:12 on dist-test-slave-9jll"
server_key: "919da45ba49d5834e3b45b0c1f4084c2"
server_key_iv: "1394a464171abe35c7218cc20cd32229"
server_key_version: "encryptionkey@0"
I20260927 18:44:12.022066 24665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:12.039388 24665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:12.045171 24665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:12.045521 24665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:12.067025 24665 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:40239
I20260927 18:44:12.067044 24731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:40239 every 8 connection(s)
I20260927 18:44:12.068816 24665 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:12.070958 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 24665
I20260927 18:44:12.071156 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:44:12.071659 2429 external_mini_cluster.cc:1428] Setting key bbb78e718eb7721ec99e7126356aaee8
I20260927 18:44:12.074640 24732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:12.090333 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:12.073836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54833 (local address 127.2.95.126:40239)
0927 18:44:12.074925 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:12.074945 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:12.075006 (+ 61us) server_negotiation.cc:407] Connection header received
0927 18:44:12.077540 (+ 2534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:12.077579 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:12.078280 (+ 701us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:12.078877 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:12.079695 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:12.081919 (+ 2224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:12.083455 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:12.084792 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:12.085019 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:12.086605 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:12.086619 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:12.086644 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:44:12.087574 (+ 930us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:12.087721 (+ 147us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:44:12.087761 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:12.087897 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:12.088175 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 18:44:12.088562 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":462,"threads_started":1}
I20260927 18:44:12.091732 24732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603: Bootstrap starting.
I20260927 18:44:12.094971 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:44:12.096050 24732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:12.097626 24732 log.cc:824] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:12.102244 24732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603: No bootstrap required, opened a new log
I20260927 18:44:12.109081 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:12.100808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51119 (local address 127.2.95.126:40239)
0927 18:44:12.100987 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:12.100992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:12.102352 (+ 1360us) server_negotiation.cc:407] Connection header received
0927 18:44:12.102657 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:12.102663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:12.102759 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:12.102887 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:12.103467 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:12.104602 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:12.106218 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:12.106939 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:12.107080 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:12.108217 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:12.108223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:12.108228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:44:12.108255 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:12.108314 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:44:12.108336 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:12.108440 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:12.108552 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 18:44:12.108673 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:44:12.112957 24732 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ac4ec96b214f1b8a3ffedd8d153603" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40239 } }
I20260927 18:44:12.113449 24732 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:12.113531 24732 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ac4ec96b214f1b8a3ffedd8d153603, State: Initialized, Role: FOLLOWER
I20260927 18:44:12.114307 24732 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ac4ec96b214f1b8a3ffedd8d153603" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40239 } }
I20260927 18:44:12.114603 24732 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:12.114786 24732 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:12.115005 24732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:12.117805 24732 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ac4ec96b214f1b8a3ffedd8d153603" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40239 } }
I20260927 18:44:12.118435 24732 leader_election.cc:302] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2ac4ec96b214f1b8a3ffedd8d153603; no voters:
I20260927 18:44:12.119076 24732 leader_election.cc:288] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:12.119225 24742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:12.119949 24742 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [term 1 LEADER]: Becoming Leader. State: Replica: d2ac4ec96b214f1b8a3ffedd8d153603, State: Running, Role: LEADER
I20260927 18:44:12.120752 24742 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ac4ec96b214f1b8a3ffedd8d153603" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40239 } }
I20260927 18:44:12.121495 24732 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:12.124840 24743 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2ac4ec96b214f1b8a3ffedd8d153603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ac4ec96b214f1b8a3ffedd8d153603" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40239 } } }
I20260927 18:44:12.125336 24743 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:12.125191 24744 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2ac4ec96b214f1b8a3ffedd8d153603. Latest consensus state: current_term: 1 leader_uuid: "d2ac4ec96b214f1b8a3ffedd8d153603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ac4ec96b214f1b8a3ffedd8d153603" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 40239 } } }
I20260927 18:44:12.125456 24744 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:12.126418 24751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:12.134263 24751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:12.147135 24751 catalog_manager.cc:1347] Generated new cluster ID: 433eb64863ef4937ba23150f4ef263b4
I20260927 18:44:12.147287 24751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:12.167208 24751 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:12.169319 24751 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:12.184149 24751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2ac4ec96b214f1b8a3ffedd8d153603: Generated new TSK 0
I20260927 18:44:12.185710 24751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:44:12.196884 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24665
W20260927 18:44:12.198365 24727 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54833
W20260927 18:44:12.199366 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24665: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:12.199590 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 24665
2026-09-27T18:44:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:12 dist-test-slave-9jll krb5kdc[24763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:12 dist-test-slave-9jll krb5kdc[24763](info): set up 2 sockets
Sep 27 18:44:12 dist-test-slave-9jll krb5kdc[24763](info): commencing operation
krb5kdc: starting...
W20260927 18:44:13.266358 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.005s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:44:13 dist-test-slave-9jll krb5kdc[24763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:44:13.581744 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33981
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:45937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:33981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:13.977746 24776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:13.978142 24776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:13.978274 24776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:13.986289 24776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:13.986424 24776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:13.986497 24776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:13.986584 24776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:13.986632 24776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:13.998446 24776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:45937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:33981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:33981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:14.000285 24776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:14.002060 24776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:14.017237 24781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:14.017256 24784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:14.017285 24782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:14.023536 24776 server_base.cc:1034] running on GCE node
I20260927 18:44:14.024729 24776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:14.026707 24776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:14.027956 24776 hybrid_clock.cc:648] HybridClock initialized: now 1790534654027879 us; error 72 us; skew 500 ppm
I20260927 18:44:14.028405 24776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:14 dist-test-slave-9jll krb5kdc[24763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:14.038841 24776 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:44:14.040867 24776 webserver.cc:460] Webserver started at http://127.2.95.126:44421/ using document root <none> and password file <none>
I20260927 18:44:14.041810 24776 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:14.041953 24776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:14.042327 24776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:14.046000 24776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "0f04ca1ac6954b18acb2475efdc3b756"
format_stamp: "Formatted at 2026-09-27 18:44:14 on dist-test-slave-9jll"
server_key: "193be219a5f0b855bf35cece893b527c"
server_key_iv: "652e8d5dc2342eed419eafb46ca3c2de"
server_key_version: "encryptionkey@0"
I20260927 18:44:14.046871 24776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "0f04ca1ac6954b18acb2475efdc3b756"
format_stamp: "Formatted at 2026-09-27 18:44:14 on dist-test-slave-9jll"
server_key: "193be219a5f0b855bf35cece893b527c"
server_key_iv: "652e8d5dc2342eed419eafb46ca3c2de"
server_key_version: "encryptionkey@0"
I20260927 18:44:14.053098 24776 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 18:44:14.057453 24791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:14.059149 24776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 18:44:14.059342 24776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "0f04ca1ac6954b18acb2475efdc3b756"
format_stamp: "Formatted at 2026-09-27 18:44:14 on dist-test-slave-9jll"
server_key: "193be219a5f0b855bf35cece893b527c"
server_key_iv: "652e8d5dc2342eed419eafb46ca3c2de"
server_key_version: "encryptionkey@0"
I20260927 18:44:14.059561 24776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:14.066195 24776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:14.070849 24776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:14.071153 24776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:14.091372 24776 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:33981
I20260927 18:44:14.091396 24842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:33981 every 8 connection(s)
I20260927 18:44:14.093051 24776 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:14.094846 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 24776
I20260927 18:44:14.095036 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:44:14.095544 2429 external_mini_cluster.cc:1428] Setting key 3311c8338fda927f951fe4e4a3117856
I20260927 18:44:14.098188 24843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:14.105626 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:14.097597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40529 (local address 127.2.95.126:33981)
0927 18:44:14.098519 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:14.098535 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:14.098585 (+ 50us) server_negotiation.cc:407] Connection header received
0927 18:44:14.100389 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:14.100416 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:14.100819 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:14.101234 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:14.102637 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:14.102652 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:14.102672 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:44:14.102693 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:14.102809 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:44:14.102844 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:14.103089 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:14.103733 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 18:44:14.104016 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":356,"threads_started":1}
I20260927 18:44:14.110175 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:44:14.112739 24843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756: Bootstrap starting.
I20260927 18:44:14.117578 24843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:14.119050 24843 log.cc:824] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:14.120658 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:14.115163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37317 (local address 127.2.95.126:33981)
0927 18:44:14.115348 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:14.115353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:14.118372 (+ 3019us) server_negotiation.cc:407] Connection header received
0927 18:44:14.118567 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:14.118571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:14.118634 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:14.118732 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:14.119812 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:14.119816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:14.119820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 18:44:14.119840 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:14.119886 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 18:44:14.119902 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:14.120011 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:14.120190 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 18:44:14.120257 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:44:14.123051 24843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756: No bootstrap required, opened a new log
I20260927 18:44:14.132138 24843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f04ca1ac6954b18acb2475efdc3b756" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33981 } }
I20260927 18:44:14.132798 24843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 18:44:14.132921 24843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f04ca1ac6954b18acb2475efdc3b756, State: Initialized, Role: FOLLOWER
I20260927 18:44:14.133703 24843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f04ca1ac6954b18acb2475efdc3b756" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33981 } }
I20260927 18:44:14.133958 24843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 18:44:14.134102 24843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 18:44:14.134289 24843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:14.136977 24843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f04ca1ac6954b18acb2475efdc3b756" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33981 } }
I20260927 18:44:14.137706 24843 leader_election.cc:302] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f04ca1ac6954b18acb2475efdc3b756; no voters:
I20260927 18:44:14.138443 24843 leader_election.cc:288] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 18:44:14.138607 24853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:14.139909 24853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [term 1 LEADER]: Becoming Leader. State: Replica: 0f04ca1ac6954b18acb2475efdc3b756, State: Running, Role: LEADER
I20260927 18:44:14.140558 24853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f04ca1ac6954b18acb2475efdc3b756" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33981 } }
I20260927 18:44:14.141043 24843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:14.143697 24854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f04ca1ac6954b18acb2475efdc3b756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f04ca1ac6954b18acb2475efdc3b756" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33981 } } }
I20260927 18:44:14.143548 24855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f04ca1ac6954b18acb2475efdc3b756. Latest consensus state: current_term: 1 leader_uuid: "0f04ca1ac6954b18acb2475efdc3b756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f04ca1ac6954b18acb2475efdc3b756" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 33981 } } }
I20260927 18:44:14.144096 24854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:14.144096 24855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:14.145207 24862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:14.151707 24862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:14.163115 24862 catalog_manager.cc:1347] Generated new cluster ID: 49840e3f79f545798f930111e881ab6f
I20260927 18:44:14.163291 24862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:14.187469 24862 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:14.188123 24862 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:14.202433 24862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f04ca1ac6954b18acb2475efdc3b756: Generated new TSK 0
I20260927 18:44:14.203596 24862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 18:44:14.217391 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24776
W20260927 18:44:14.219998 24838 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40529
W20260927 18:44:14.221397 2429 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24776: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 18:44:14.221585 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 24776
2026-09-27T18:44:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2026 ms)
[----------] 18 tests from AuthTokenIssuingTest (37095 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:14 dist-test-slave-9jll krb5kdc[24874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:14 dist-test-slave-9jll krb5kdc[24874](info): set up 2 sockets
Sep 27 18:44:14 dist-test-slave-9jll krb5kdc[24874](info): commencing operation
krb5kdc: starting...
W20260927 18:44:15.286495 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.011s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:44:15 dist-test-slave-9jll krb5kdc[24874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:44:15.609588 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:45031
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:45031,127.2.95.125:37525,127.2.95.124:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:15.998297 24887 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:44:15.998795 24887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:15.998904 24887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:15.999038 24887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:16.007040 24887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:16.007191 24887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:16.007280 24887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:16.007359 24887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:16.007452 24887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:16.019024 24887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:45031,127.2.95.125:37525,127.2.95.124:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:45031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:16.020926 24887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:16.022742 24887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:16.037097 24892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:16.036978 24895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:16.037006 24893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:16.037802 24887 server_base.cc:1034] running on GCE node
I20260927 18:44:16.038816 24887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:16.040701 24887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:16.041947 24887 hybrid_clock.cc:648] HybridClock initialized: now 1790534656041874 us; error 48 us; skew 500 ppm
I20260927 18:44:16.042339 24887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:16 dist-test-slave-9jll krb5kdc[24874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:16.046207 24887 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:44:16.048081 24887 webserver.cc:460] Webserver started at http://127.2.95.126:37211/ using document root <none> and password file <none>
I20260927 18:44:16.048945 24887 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:16.049047 24887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:16.049365 24887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:16.053211 24887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "a572340f833e4872ac6e67f43f2199bd"
format_stamp: "Formatted at 2026-09-27 18:44:16 on dist-test-slave-9jll"
server_key: "4ab0b829653ad9f5e79a4640a6d70cec"
server_key_iv: "cb96774816ea775f0ecb1be33500d667"
server_key_version: "encryptionkey@0"
I20260927 18:44:16.054159 24887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "a572340f833e4872ac6e67f43f2199bd"
format_stamp: "Formatted at 2026-09-27 18:44:16 on dist-test-slave-9jll"
server_key: "4ab0b829653ad9f5e79a4640a6d70cec"
server_key_iv: "cb96774816ea775f0ecb1be33500d667"
server_key_version: "encryptionkey@0"
I20260927 18:44:16.060504 24887 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:44:16.064741 24902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:16.066674 24887 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:44:16.066876 24887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "a572340f833e4872ac6e67f43f2199bd"
format_stamp: "Formatted at 2026-09-27 18:44:16 on dist-test-slave-9jll"
server_key: "4ab0b829653ad9f5e79a4640a6d70cec"
server_key_iv: "cb96774816ea775f0ecb1be33500d667"
server_key_version: "encryptionkey@0"
I20260927 18:44:16.067128 24887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:16.090116 24887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:16.095491 24887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:16.095777 24887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:16.117852 24887 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:45031
I20260927 18:44:16.117844 24953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:45031 every 8 connection(s)
I20260927 18:44:16.119618 24887 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:16.123426 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 24887
I20260927 18:44:16.123634 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:44:16.124367 2429 external_mini_cluster.cc:1428] Setting key 609a92034f10f3dfcdb06c6a8cfd26c6
I20260927 18:44:16.124572 24954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:16.132589 24954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:16 dist-test-slave-9jll krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.95.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.125@KRBTEST.COM" created.
I20260927 18:44:16.160795 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.138746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50088 (local address 127.2.95.126:45031)
0927 18:44:16.139550 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:16.139560 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:16.139649 (+ 89us) server_negotiation.cc:407] Connection header received
0927 18:44:16.141130 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:16.141137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:16.141440 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:16.141852 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:16.143401 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.144889 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.146538 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.147165 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.147368 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:16.151558 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:16.151568 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:16.151586 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:16.151620 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:16.155077 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:16.155906 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:16.155915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:16.155918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:16.156133 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:16.156561 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:16.156566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:16.156569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:16.157424 (+ 855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:16.157729 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:16.158552 (+ 823us) server_negotiation.cc:299] Negotiation successful
0927 18:44:16.159102 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":210,"server-negotiator.queue_time_us":560,"thread_start_us":366,"threads_started":1}
I20260927 18:44:16.160822 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.137061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45031 (local address 127.0.0.1:50088)
0927 18:44:16.138160 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.138213 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:16.140666 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:16.141871 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:16.141878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:16.142766 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:16.143177 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:16.143199 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.145078 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.145082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:16.146394 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:16.146408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.147420 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.147426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:16.147641 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:16.148374 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:16.148401 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:16.151247 (+ 2846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:16.155260 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:16.155276 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:16.155294 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:16.155744 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:16.156253 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:16.156256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:16.156259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:16.156443 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:16.157587 (+ 1144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:16.157597 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:16.157916 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 18:44:16.158297 (+ 381us) client_negotiation.cc:240] Negotiation successful
0927 18:44:16.158923 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":505,"threads_started":1}
I20260927 18:44:16.165303 24954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:44:16.166980 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.166400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35190)
0927 18:44:16.166676 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.166819 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260927 18:44:16.167347 24903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
I20260927 18:44:16.169121 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.168696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35204)
0927 18:44:16.168880 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.168960 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 18:44:16.169332 24954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:37525: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.125.keytab.
Entry for principal kudu/127.2.95.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.125.keytab.
I20260927 18:44:16.194355 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.125:37525
--webserver_interface=127.2.95.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.125.keytab
--principal=kudu/127.2.95.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:45031,127.2.95.125:37525,127.2.95.124:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:44:16.218828 24954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } attempt: 1
I20260927 18:44:16.220458 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.219874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35212)
0927 18:44:16.220147 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.220260 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 18:44:16.221853 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.221349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35216)
0927 18:44:16.221523 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.221607 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 18:44:16.222059 24954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:37525: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
I20260927 18:44:16.290378 24954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } attempt: 2
I20260927 18:44:16.291850 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.291279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35222)
0927 18:44:16.291565 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.291679 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 18:44:16.292989 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.292612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35232)
0927 18:44:16.292764 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.292835 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 18:44:16.293182 24954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:37525: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
I20260927 18:44:16.384476 24954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } attempt: 3
I20260927 18:44:16.385953 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.385378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35248)
0927 18:44:16.385623 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.385778 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 18:44:16.387223 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.386795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35264)
0927 18:44:16.386958 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.387058 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 18:44:16.387431 24954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:37525: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
I20260927 18:44:16.530719 24954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } attempt: 4
I20260927 18:44:16.532200 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.531567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35266)
0927 18:44:16.531906 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.532035 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260927 18:44:16.533533 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.533107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35278)
0927 18:44:16.533270 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.533361 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 18:44:16.533763 24954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:37525: Network error: Client connection negotiation failed: client connection to 127.2.95.125:37525: connect: Connection refused (error 111)
W20260927 18:44:16.590044 24961 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:44:16.590499 24961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:16.590602 24961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:16.590729 24961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:16.598788 24961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:16.598920 24961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:16.599004 24961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:16.599076 24961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:16.599175 24961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:16.610668 24961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:45031,127.2.95.125:37525,127.2.95.124:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.125.keytab
--principal=kudu/127.2.95.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.125:37525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.95.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:16.612499 24961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:16.614319 24961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:16.628156 24966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:16.628275 24969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:16.628315 24967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:16.629792 24961 server_base.cc:1034] running on GCE node
I20260927 18:44:16.630798 24961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:16.632651 24961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:16.633908 24961 hybrid_clock.cc:648] HybridClock initialized: now 1790534656633822 us; error 54 us; skew 500 ppm
I20260927 18:44:16.634302 24961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:16 dist-test-slave-9jll krb5kdc[24874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:16.638057 24961 init.cc:377] Logged in from keytab as kudu/127.2.95.125@KRBTEST.COM (short username kudu)
I20260927 18:44:16.640004 24961 webserver.cc:460] Webserver started at http://127.2.95.125:34551/ using document root <none> and password file <none>
I20260927 18:44:16.640883 24961 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:16.640991 24961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:16.641315 24961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:16.644984 24961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/data/instance:
uuid: "1112378a652c4b6e8376cdc5d06f41d6"
format_stamp: "Formatted at 2026-09-27 18:44:16 on dist-test-slave-9jll"
server_key: "7f5de55054ae2b9e0a43b6421c843424"
server_key_iv: "e7d4b7e34ad7f3ab6abb0291bf51fcb4"
server_key_version: "encryptionkey@0"
I20260927 18:44:16.645924 24961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/wal/instance:
uuid: "1112378a652c4b6e8376cdc5d06f41d6"
format_stamp: "Formatted at 2026-09-27 18:44:16 on dist-test-slave-9jll"
server_key: "7f5de55054ae2b9e0a43b6421c843424"
server_key_iv: "e7d4b7e34ad7f3ab6abb0291bf51fcb4"
server_key_version: "encryptionkey@0"
I20260927 18:44:16.652521 24961 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 18:44:16.657423 24976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:16.659466 24961 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:44:16.659706 24961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/wal
uuid: "1112378a652c4b6e8376cdc5d06f41d6"
format_stamp: "Formatted at 2026-09-27 18:44:16 on dist-test-slave-9jll"
server_key: "7f5de55054ae2b9e0a43b6421c843424"
server_key_iv: "e7d4b7e34ad7f3ab6abb0291bf51fcb4"
server_key_version: "encryptionkey@0"
I20260927 18:44:16.659981 24961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:16.690331 24961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:16.695879 24961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:16.696236 24961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:16.717443 24961 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.125:37525
I20260927 18:44:16.717451 25027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.125:37525 every 8 connection(s)
I20260927 18:44:16.719120 24961 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
I20260927 18:44:16.720427 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 24961
I20260927 18:44:16.720721 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-1/wal/instance
I20260927 18:44:16.721328 2429 external_mini_cluster.cc:1428] Setting key 5577cf7a7e8401b420699c6836ae1e0e
I20260927 18:44:16.724864 25028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:16.735282 25028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:16 dist-test-slave-9jll krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.125@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.95.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.124@KRBTEST.COM" created.
I20260927 18:44:16.762290 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.738716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.2.95.126:45031)
0927 18:44:16.739356 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:16.739363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:16.740648 (+ 1285us) server_negotiation.cc:407] Connection header received
0927 18:44:16.743044 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:16.743052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:16.743275 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:16.743423 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:16.745439 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.746551 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.748583 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.749262 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.749399 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:16.753871 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:16.753880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:16.753885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:16.753958 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:16.757631 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:16.759077 (+ 1446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:16.759083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:16.759086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:16.759263 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:16.759884 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:16.759888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:16.759891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:16.760297 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:16.760440 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:16.761538 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0927 18:44:16.761844 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":301,"threads_started":1}
I20260927 18:44:16.764019 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.739466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45031 (local address 127.0.0.1:50094)
0927 18:44:16.740431 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.740487 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:16.742714 (+ 2227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:16.743706 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:16.743717 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:16.744436 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:16.745165 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:16.745186 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.746784 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.746789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:16.748411 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:16.748427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.749547 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.749553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:16.749929 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:16.750765 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:16.750794 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:16.753469 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:16.757944 (+ 4475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:16.757960 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:16.757987 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:16.758854 (+ 867us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:16.759391 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:16.759397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:16.759401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:16.759727 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:16.760445 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:16.760456 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:16.761080 (+ 624us) client_negotiation.cc:769] Sending connection context
0927 18:44:16.761431 (+ 351us) client_negotiation.cc:240] Negotiation successful
0927 18:44:16.762230 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":394,"threads_started":1}
I20260927 18:44:16.766387 25028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:16 dist-test-slave-9jll krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.125@KRBTEST.COM for kudu/127.2.95.125@KRBTEST.COM
I20260927 18:44:16.786080 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.767318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35292)
0927 18:44:16.767529 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.767547 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:16.767655 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:16.771462 (+ 3807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:16.771467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:16.772052 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:16.772239 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:16.772249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.773876 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.773881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:16.774793 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:16.774806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.775727 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.775733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:16.775896 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:16.776580 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:16.776600 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:16.779135 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:16.783253 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:16.783259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:16.783263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:16.783689 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:16.784114 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:16.784118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:16.784120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:16.784268 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:16.784964 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:16.784969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:16.785118 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 18:44:16.785243 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 18:44:16.785589 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 18:44:16.786141 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.770060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.2.95.125:37525)
0927 18:44:16.770903 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:16.770911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:16.770944 (+ 33us) server_negotiation.cc:407] Connection header received
0927 18:44:16.771010 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:16.771017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:16.771273 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:16.771530 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:16.772413 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.773740 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.774929 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.775551 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.775677 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:16.779368 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:16.779383 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:16.779404 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:16.779439 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:16.783075 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:16.783829 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:16.783838 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:16.783840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:16.783969 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:16.784374 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:16.784379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:16.784382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:16.784826 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:16.785034 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:16.785411 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 18:44:16.785773 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":660,"thread_start_us":456,"threads_started":1}
I20260927 18:44:16.789022 25028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:44:16.790583 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.789960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59538)
0927 18:44:16.790207 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.790400 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 18:44:16.790874 24978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
I20260927 18:44:16.792541 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.792147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59548)
0927 18:44:16.792308 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.792400 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 18:44:16.792804 25028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:42629: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.124.keytab.
Entry for principal kudu/127.2.95.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.124.keytab.
I20260927 18:44:16.798444 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.124:42629
--webserver_interface=127.2.95.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.124.keytab
--principal=kudu/127.2.95.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:38785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:45031,127.2.95.125:37525,127.2.95.124:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:44:16.833048 24954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } attempt: 5
Sep 27 18:44:16 dist-test-slave-9jll krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for kudu/127.2.95.125@KRBTEST.COM
I20260927 18:44:16.842191 25028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } attempt: 1
I20260927 18:44:16.843456 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.842909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59558)
0927 18:44:16.843135 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.843251 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 18:44:16.844767 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.844347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59560)
0927 18:44:16.844544 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.844633 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 18:44:16.845057 25028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:42629: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
I20260927 18:44:16.849200 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.834330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.2.95.125:37525)
0927 18:44:16.834577 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:16.834584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:16.834605 (+ 21us) server_negotiation.cc:407] Connection header received
0927 18:44:16.834677 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:16.834681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:16.834755 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:16.834845 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:16.835799 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.836580 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.837939 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.838426 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.838556 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:16.842598 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:16.842606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:16.842611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:16.842661 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:16.846110 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:16.846983 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:16.846989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:16.846991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:16.847153 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:16.847610 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:16.847616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:16.847619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:16.847963 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:16.848129 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:16.848712 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 18:44:16.848915 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:44:16.849424 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.834064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35302)
0927 18:44:16.834296 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.834317 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:16.834459 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:16.834918 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:16.834923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:16.835399 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:16.835625 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:16.835637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.836729 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.836734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:16.837776 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:16.837791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:16.838589 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:16.838595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:16.838801 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:16.839592 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:16.839614 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:16.842320 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:16.846265 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:16.846270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:16.846275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:16.846846 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:16.847260 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:16.847264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:16.847268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:16.847449 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:16.848151 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:16.848157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:16.848353 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 18:44:16.848571 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 18:44:16.848922 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 18:44:16.850726 24954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:44:16.852226 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.851691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59572)
0927 18:44:16.851899 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.852008 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 18:44:16.853356 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.852916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59588)
0927 18:44:16.853089 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.853191 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 18:44:16.853588 24954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:42629: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
I20260927 18:44:16.904917 24954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } attempt: 1
I20260927 18:44:16.906795 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.905948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59596)
0927 18:44:16.906337 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.906494 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260927 18:44:16.908212 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.907754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59604)
0927 18:44:16.907929 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.908033 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 18:44:16.908468 24954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:42629: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
I20260927 18:44:16.913323 25028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } attempt: 2
I20260927 18:44:16.914767 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.914224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59616)
0927 18:44:16.914447 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.914554 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 18:44:16.916005 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.915595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59630)
0927 18:44:16.915752 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.915837 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 18:44:16.916311 25028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:42629: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
I20260927 18:44:16.976752 24954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } attempt: 2
I20260927 18:44:16.978578 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.977813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59646)
0927 18:44:16.978129 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.978304 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 18:44:16.979790 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:16.979367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59656)
0927 18:44:16.979525 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:16.979621 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 18:44:16.980054 24954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:42629: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
I20260927 18:44:17.007596 25028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } attempt: 3
I20260927 18:44:17.009430 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.008532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59658)
0927 18:44:17.008885 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.009183 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260927 18:44:17.010785 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.010235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59674)
0927 18:44:17.010539 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.010635 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
W20260927 18:44:17.011049 25028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:42629: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
I20260927 18:44:17.086309 24954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } attempt: 3
I20260927 18:44:17.088016 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.087238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59678)
0927 18:44:17.087551 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.087733 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 18:44:17.089298 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.088819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59688)
0927 18:44:17.089065 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.089151 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260927 18:44:17.089581 24954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:42629: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
I20260927 18:44:17.154292 25028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } attempt: 4
I20260927 18:44:17.155826 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.155245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59696)
0927 18:44:17.155521 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.155632 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 18:44:17.156972 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.156603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59706)
0927 18:44:17.156761 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.156844 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 18:44:17.157186 25028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:42629: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
W20260927 18:44:17.208271 25036 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:44:17.208732 25036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:17.208833 25036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:17.208952 25036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:17.217258 25036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:17.217407 25036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:17.217495 25036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:17.217567 25036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:17.217659 25036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:17.228982 25036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:38785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:45031,127.2.95.125:37525,127.2.95.124:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.124.keytab
--principal=kudu/127.2.95.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.124:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.95.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:17.231055 25036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:17.232788 25036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:17.246905 25041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:17.246966 25042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:17.247016 25044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:17.248296 25036 server_base.cc:1034] running on GCE node
I20260927 18:44:17.249325 25036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:17.251246 25036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:17.252614 25036 hybrid_clock.cc:648] HybridClock initialized: now 1790534657252531 us; error 43 us; skew 500 ppm
I20260927 18:44:17.253006 25036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:17 dist-test-slave-9jll krb5kdc[24874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:17.257095 25036 init.cc:377] Logged in from keytab as kudu/127.2.95.124@KRBTEST.COM (short username kudu)
I20260927 18:44:17.258998 25036 webserver.cc:460] Webserver started at http://127.2.95.124:39415/ using document root <none> and password file <none>
I20260927 18:44:17.259872 25036 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:17.259970 25036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:17.260296 25036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:17.264074 25036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/data/instance:
uuid: "8dfdc3305e4a461e804e74753ac20a49"
format_stamp: "Formatted at 2026-09-27 18:44:17 on dist-test-slave-9jll"
server_key: "f7958aaa6e319fcd2bc6b1ced09f4aeb"
server_key_iv: "d7529a996f763fad6a4777d812d2003d"
server_key_version: "encryptionkey@0"
I20260927 18:44:17.264990 25036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/wal/instance:
uuid: "8dfdc3305e4a461e804e74753ac20a49"
format_stamp: "Formatted at 2026-09-27 18:44:17 on dist-test-slave-9jll"
server_key: "f7958aaa6e319fcd2bc6b1ced09f4aeb"
server_key_iv: "d7529a996f763fad6a4777d812d2003d"
server_key_version: "encryptionkey@0"
I20260927 18:44:17.266846 24954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } attempt: 4
I20260927 18:44:17.268224 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.267672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59710)
0927 18:44:17.267912 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.268055 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 18:44:17.269411 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.268992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59720)
0927 18:44:17.269180 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.269262 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260927 18:44:17.269660 24954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:42629: Network error: Client connection negotiation failed: client connection to 127.2.95.124:42629: connect: Connection refused (error 111)
I20260927 18:44:17.271993 25036 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260927 18:44:17.276628 25051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:17.278591 25036 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 18:44:17.278800 25036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/wal
uuid: "8dfdc3305e4a461e804e74753ac20a49"
format_stamp: "Formatted at 2026-09-27 18:44:17 on dist-test-slave-9jll"
server_key: "f7958aaa6e319fcd2bc6b1ced09f4aeb"
server_key_iv: "d7529a996f763fad6a4777d812d2003d"
server_key_version: "encryptionkey@0"
I20260927 18:44:17.279076 25036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:17.300704 25036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:17.306005 25036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:17.306448 25036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:17.327113 25036 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.124:42629
I20260927 18:44:17.327124 25102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.124:42629 every 8 connection(s)
I20260927 18:44:17.328776 25036 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
I20260927 18:44:17.333220 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 25036
I20260927 18:44:17.333417 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790534284715420-2429-0/minicluster-data/master-2/wal/instance
I20260927 18:44:17.333844 25103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:17.334008 2429 external_mini_cluster.cc:1428] Setting key ddbfa080441bb5e701ec9be4fab560c1
I20260927 18:44:17.344462 25103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:17 dist-test-slave-9jll krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.124@KRBTEST.COM
Sep 27 18:44:17 dist-test-slave-9jll krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.124@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:44:17.362462 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.336359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.2.95.124:42629)
0927 18:44:17.337311 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:17.337326 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:17.337377 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:44:17.339793 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:17.339821 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:17.340451 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:17.341018 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:17.342788 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.344208 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.345585 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.346702 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.346913 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.352776 (+ 5863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:17.352788 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:17.352805 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:17.352863 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:17.356920 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.358047 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.358058 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.358070 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.358349 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.358958 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.358963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.358965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.359686 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:17.360016 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:17.360508 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 18:44:17.361052 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":442,"threads_started":1}
I20260927 18:44:17.365150 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.347945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.2.95.126:45031)
0927 18:44:17.348569 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:17.348576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:17.349097 (+ 521us) server_negotiation.cc:407] Connection header received
0927 18:44:17.349374 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:17.349380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:17.349479 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:17.349695 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:17.350984 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.352008 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.353978 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.354427 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.354532 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.358951 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:17.358956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:17.358960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:17.359011 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:17.361982 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.362803 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.362809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.362812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.362991 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.363394 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.363399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.363402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.363799 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:17.363986 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:17.364496 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 18:44:17.364746 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":305,"threads_started":1}
I20260927 18:44:17.365393 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.347930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:45031 (local address 127.0.0.1:50098)
0927 18:44:17.348895 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.348967 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:17.349181 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:17.349792 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:17.349799 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:17.350548 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:17.350825 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:17.350834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.352202 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.352206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:17.353806 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:17.353820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.354625 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.354630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:17.354805 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.355385 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:17.355456 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:17.358736 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:17.362121 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:17.362133 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:17.362141 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:17.362612 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:17.363115 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:17.363119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:17.363121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:17.363275 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:17.363949 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:17.363958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:17.364155 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 18:44:17.364374 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 18:44:17.364766 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":411,"threads_started":1}
I20260927 18:44:17.367713 25103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:17 dist-test-slave-9jll krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.124@KRBTEST.COM for kudu/127.2.95.125@KRBTEST.COM
Sep 27 18:44:17 dist-test-slave-9jll krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:44:17.383224 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.368426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:37525 (local address 127.0.0.1:35308)
0927 18:44:17.368620 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.368642 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:17.368746 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:17.369468 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:17.369472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:17.369928 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:17.370101 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:17.370110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.371185 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.371191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:17.372188 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:17.372200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.373198 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.373205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:17.373380 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.373954 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:17.373974 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:17.376351 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:17.380073 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:17.380079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:17.380084 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:17.380663 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:17.381096 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:17.381100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:17.381103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:17.381269 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:17.381947 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:17.381952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:17.382221 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 18:44:17.382409 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 18:44:17.382796 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 18:44:17.383687 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.368582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.2.95.125:37525)
0927 18:44:17.369155 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:17.369160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:17.369176 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:44:17.369243 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:17.369248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:17.369315 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:17.369424 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:17.370313 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.371039 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.372355 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.372992 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.373094 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.376601 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:17.376610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:17.376616 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:17.376666 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:17.379879 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.380785 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.380792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.380796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.380962 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.381383 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.381387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.381389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.381804 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:17.381974 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:17.382585 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 18:44:17.383058 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":330,"threads_started":1}
I20260927 18:44:17.384508 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.366863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.2.95.126:45031)
0927 18:44:17.367033 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:17.367038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:17.367057 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:44:17.367335 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:17.367340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:17.367434 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:17.367628 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:17.369469 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.370323 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.371696 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.372176 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.372313 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.377304 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:17.377310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:17.377314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:17.377355 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:17.380410 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.381302 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.381309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.381313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.381471 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.381984 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.381989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.381992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.382442 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:17.382842 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:17.383176 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 18:44:17.384159 (+ 983us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:44:17.385106 25103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:17 dist-test-slave-9jll krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.124@KRBTEST.COM for kudu/127.2.95.124@KRBTEST.COM
Sep 27 18:44:17 dist-test-slave-9jll krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.125@KRBTEST.COM
I20260927 18:44:17.400911 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.386643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59738 (local address 127.2.95.124:42629)
0927 18:44:17.386854 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:17.386860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:17.386882 (+ 22us) server_negotiation.cc:407] Connection header received
0927 18:44:17.387091 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:17.387098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:17.387187 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:17.387296 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:17.388207 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.388970 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.390277 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.390710 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.390810 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.394601 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:17.394608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:17.394614 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:17.394656 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:17.397559 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.398398 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.398403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.398406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.398538 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.399008 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.399014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.399018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.399418 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:17.399545 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:17.400052 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 18:44:17.400638 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":228,"server-negotiator.queue_time_us":98}
I20260927 18:44:17.400938 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.386462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59738)
0927 18:44:17.386773 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.386791 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:17.386893 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:17.387312 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:17.387317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:17.387802 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:17.388023 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:17.388037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.389111 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.389116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:17.390127 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:17.390139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.390880 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.390887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:17.391099 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.391773 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:17.391794 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:17.394397 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:17.397733 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:17.397739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:17.397744 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:17.398266 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:17.398668 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:17.398674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:17.398678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:17.398864 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:17.399554 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:17.399559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:17.399773 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 18:44:17.399950 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 18:44:17.400382 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":5}
I20260927 18:44:17.404246 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.387340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35324 (local address 127.2.95.125:37525)
0927 18:44:17.387524 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:17.387528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:17.387544 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:44:17.387592 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:17.387596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:17.387659 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:17.387737 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:17.389264 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.390097 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.391714 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.392138 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.392241 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.397285 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:17.397292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:17.397295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:17.397339 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:17.400830 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.401915 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.401922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.401925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.402112 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.402531 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.402537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.402540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.402913 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:17.403152 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:17.403634 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 18:44:17.403879 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:44:17.407648 25103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49: Bootstrap starting.
I20260927 18:44:17.410215 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:44:17.411999 25103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:17.413445 25103 log.cc:824] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:17.413329 24929 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a572340f833e4872ac6e67f43f2199bd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50114
I20260927 18:44:17.417491 25103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49: No bootstrap required, opened a new log
I20260927 18:44:17.429911 25103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } }
I20260927 18:44:17.430539 25103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfdc3305e4a461e804e74753ac20a49, State: Initialized, Role: FOLLOWER
I20260927 18:44:17.431205 25103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } }
I20260927 18:44:17.432384 25111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } } }
I20260927 18:44:17.432765 25111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:17.434170 25103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:44:17.446105 25122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:44:17.446293 25122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:44:17.442793 25036 master_runner.cc:186] Error getting master registration for 127.2.95.126:45031: OK, instance_id { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" instance_seqno: 1790534656110881 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:44:17.455039 25036 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:44:17.456378 25028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } attempt: 5
Sep 27 18:44:17 dist-test-slave-9jll krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.125@KRBTEST.COM for kudu/127.2.95.124@KRBTEST.COM
I20260927 18:44:17.473995 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.457228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59742)
0927 18:44:17.457475 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.457501 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:17.457702 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:17.458227 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:17.458234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:17.458807 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:17.459006 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:17.459018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.460197 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.460203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:17.461124 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:17.461138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.462024 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.462031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:17.462281 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.463028 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:17.463057 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:17.466185 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:17.470864 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:17.470871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:17.470876 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:17.471450 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:17.471916 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:17.471922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:17.471926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:17.472145 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:17.472818 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:17.472823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:17.473005 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 18:44:17.473174 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 18:44:17.473541 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 18:44:17.473995 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.457477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.2.95.124:42629)
0927 18:44:17.457764 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:17.457771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:17.457795 (+ 24us) server_negotiation.cc:407] Connection header received
0927 18:44:17.457877 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:17.457881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:17.457954 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:17.458124 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:17.459203 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.460030 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.461271 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.461861 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.461977 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.466419 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:17.466427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:17.466433 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:17.466480 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:17.470687 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.471601 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.471608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.471612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.471782 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.472257 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.472262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.472266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.472696 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:17.472862 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:17.473327 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 18:44:17.473568 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:44:17.481609 25028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6: Bootstrap starting.
I20260927 18:44:17.486836 25028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:17.488559 25028 log.cc:824] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:17.492934 25028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6: No bootstrap required, opened a new log
I20260927 18:44:17.505208 25028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } }
I20260927 18:44:17.505767 25028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1112378a652c4b6e8376cdc5d06f41d6, State: Initialized, Role: FOLLOWER
I20260927 18:44:17.506423 25028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } }
I20260927 18:44:17.507951 25125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } } }
I20260927 18:44:17.508776 25125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:17.509513 25028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:44:17.521838 25136 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:44:17.522078 25136 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:44:17.518352 24961 master_runner.cc:186] Error getting master registration for 127.2.95.126:45031: OK, instance_id { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" instance_seqno: 1790534656110881 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:44:17.539047 24961 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:44:17.546895 24954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } attempt: 5
Sep 27 18:44:17 dist-test-slave-9jll krb5kdc[24874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534656, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for kudu/127.2.95.124@KRBTEST.COM
W20260927 18:44:17.559226 25057 tablet.cc:2369] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:17.568150 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.548085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:42629 (local address 127.0.0.1:59752)
0927 18:44:17.548492 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:17.548516 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:17.548682 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:17.549341 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:17.549349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:17.550077 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:17.550369 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:17.550385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.551994 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.552001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:17.553614 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:17.553633 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.554756 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.554763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:17.554986 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.556000 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:17.556029 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:17.559442 (+ 3413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:17.564295 (+ 4853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:17.564302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:17.564307 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:17.565116 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:17.565746 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:17.565752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:17.565756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:17.566003 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:17.566920 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:17.566926 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:17.567129 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 18:44:17.567284 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 18:44:17.567664 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260927 18:44:17.569527 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.548198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.2.95.124:42629)
0927 18:44:17.548443 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:17.548450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:17.548607 (+ 157us) server_negotiation.cc:407] Connection header received
0927 18:44:17.548866 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:17.548873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:17.549098 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:17.549232 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:17.550565 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.551810 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.553818 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.554566 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.554709 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.559761 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:17.559771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:17.559776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:17.559829 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:17.564058 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.565300 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.565307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.565311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.565548 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.566155 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.566162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.566165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.566689 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:17.566850 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:17.568644 (+ 1794us) server_negotiation.cc:299] Negotiation successful
0927 18:44:17.568937 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 18:44:17.577030 24954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd: Bootstrap starting.
I20260927 18:44:17.582432 24954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:17.584131 24954 log.cc:824] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:17.590090 24954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd: No bootstrap required, opened a new log
I20260927 18:44:17.603565 24954 raft_consensus.cc:348] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } }
I20260927 18:44:17.604390 24954 raft_consensus.cc:729] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a572340f833e4872ac6e67f43f2199bd, State: Initialized, Role: FOLLOWER
I20260927 18:44:17.605355 24954 consensus_queue.cc:260] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } }
I20260927 18:44:17.606905 25139 sys_catalog.cc:455] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } } }
I20260927 18:44:17.607461 25139 sys_catalog.cc:458] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:17.609967 24929 raft_consensus.cc:482] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 18:44:17.610225 24929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:17.610526 24954 sys_catalog.cc:564] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:17.614475 24929 raft_consensus.cc:504] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } }
I20260927 18:44:17.618012 25003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a572340f833e4872ac6e67f43f2199bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1112378a652c4b6e8376cdc5d06f41d6"
I20260927 18:44:17.618654 25003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:17.623234 25003 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a572340f833e4872ac6e67f43f2199bd in term 1.
I20260927 18:44:17.624982 24903 leader_election.cc:302] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1112378a652c4b6e8376cdc5d06f41d6, a572340f833e4872ac6e67f43f2199bd; no voters:
I20260927 18:44:17.625854 25139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:17.631353 25078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a572340f833e4872ac6e67f43f2199bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8dfdc3305e4a461e804e74753ac20a49"
I20260927 18:44:17.633373 25078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:17.638203 25078 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a572340f833e4872ac6e67f43f2199bd in term 1.
I20260927 18:44:17.639968 24929 leader_election.cc:288] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [CANDIDATE]: Term 1 election: Requested vote from peers 1112378a652c4b6e8376cdc5d06f41d6 (127.2.95.125:37525), 8dfdc3305e4a461e804e74753ac20a49 (127.2.95.124:42629)
I20260927 18:44:17.661749 25139 raft_consensus.cc:686] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [term 1 LEADER]: Becoming Leader. State: Replica: a572340f833e4872ac6e67f43f2199bd, State: Running, Role: LEADER
I20260927 18:44:17.663197 25139 consensus_queue.cc:237] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } }
I20260927 18:44:17.674283 25141 sys_catalog.cc:455] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a572340f833e4872ac6e67f43f2199bd. Latest consensus state: current_term: 1 leader_uuid: "a572340f833e4872ac6e67f43f2199bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } } }
I20260927 18:44:17.674989 25141 sys_catalog.cc:458] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:17.683719 25151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 18:44:17.699631 24982 tablet.cc:2369] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:17.716940 25151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:17.725925 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.684153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.2.95.126:45031)
0927 18:44:17.684368 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:17.684377 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:17.694638 (+ 10261us) server_negotiation.cc:407] Connection header received
0927 18:44:17.698183 (+ 3545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:17.698191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:17.698302 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:17.698391 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:17.703288 (+ 4897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.704410 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.706969 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.707622 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.707715 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.715739 (+ 8024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:17.715748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:17.715754 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:17.715857 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:17.720086 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.721836 (+ 1750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.721843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.721847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.722212 (+ 365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.723265 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.723271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.723275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.723799 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:17.724156 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:17.725175 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0927 18:44:17.725474 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 18:44:17.748154 25162 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:44:17.756058 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.695365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.2.95.124:42629)
0927 18:44:17.695605 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:17.695611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:17.703411 (+ 7800us) server_negotiation.cc:407] Connection header received
0927 18:44:17.703685 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:17.703692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:17.703796 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:17.703877 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:17.709009 (+ 5132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.710289 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.717319 (+ 7030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.718051 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.718186 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.739517 (+ 21331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:17.739527 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:17.739532 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:17.739605 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:17.743932 (+ 4327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.751433 (+ 7501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.751446 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.751450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.751671 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.753385 (+ 1714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.753391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.753394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.753906 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:17.754194 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:17.755240 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0927 18:44:17.755597 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:44:17.758621 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:17.684456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.2.95.125:37525)
0927 18:44:17.684643 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:17.684649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:17.702885 (+ 18236us) server_negotiation.cc:407] Connection header received
0927 18:44:17.706539 (+ 3654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:17.706545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:17.706645 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:17.706724 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:17.712440 (+ 5716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.713710 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.724745 (+ 11035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:17.725528 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:17.725654 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:17.745480 (+ 19826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:17.745489 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:17.745495 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:17.745555 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:17.749599 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.750734 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.750741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.750745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.750912 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:17.752262 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:17.752272 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:17.752276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:17.752922 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:17.753265 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:17.757889 (+ 4624us) server_negotiation.cc:299] Negotiation successful
0927 18:44:17.758200 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 18:44:17.772693 25003 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [term 1 FOLLOWER]: Refusing update from remote peer a572340f833e4872ac6e67f43f2199bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:17.776453 25078 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [term 1 FOLLOWER]: Refusing update from remote peer a572340f833e4872ac6e67f43f2199bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:17.779182 25139 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:17.781523 25139 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:17.812747 25125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a572340f833e4872ac6e67f43f2199bd. Latest consensus state: current_term: 1 leader_uuid: "a572340f833e4872ac6e67f43f2199bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } } }
I20260927 18:44:17.813189 25125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:17.817070 25111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a572340f833e4872ac6e67f43f2199bd. Latest consensus state: current_term: 1 leader_uuid: "a572340f833e4872ac6e67f43f2199bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } } }
I20260927 18:44:17.817479 25111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:17.819348 25159 mvcc.cc:205] Tried to move back new op lower bound from 7334029958197751808 to 7334029957812600832. Current Snapshot: MvccSnapshot[applied={T|T < 7334029958197751808}]
I20260927 18:44:17.827584 25164 mvcc.cc:205] Tried to move back new op lower bound from 7334029958197751808 to 7334029957812600832. Current Snapshot: MvccSnapshot[applied={T|T < 7334029958197751808}]
I20260927 18:44:17.844741 25125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a572340f833e4872ac6e67f43f2199bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } } }
I20260927 18:44:17.845247 25125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:17.858695 25141 sys_catalog.cc:455] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a572340f833e4872ac6e67f43f2199bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } } }
I20260927 18:44:17.859098 25141 sys_catalog.cc:458] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:17.859562 25141 sys_catalog.cc:455] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a572340f833e4872ac6e67f43f2199bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } } }
I20260927 18:44:17.859802 25141 sys_catalog.cc:458] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:17.862609 25151 catalog_manager.cc:1347] Generated new cluster ID: bd20bbd8ae4e4ec89506826cde903702
I20260927 18:44:17.862732 25151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:17.863104 25111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a572340f833e4872ac6e67f43f2199bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572340f833e4872ac6e67f43f2199bd" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 45031 } } peers { permanent_uuid: "1112378a652c4b6e8376cdc5d06f41d6" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 37525 } } peers { permanent_uuid: "8dfdc3305e4a461e804e74753ac20a49" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 42629 } } }
I20260927 18:44:17.863385 25111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:17.894325 25151 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:17.899623 25151 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:17.928071 25151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a572340f833e4872ac6e67f43f2199bd: Generated new TSK 0
I20260927 18:44:17.929831 25151 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:44:18.066979 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:18.051235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.2.95.125:37525)
0927 18:44:18.051543 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.051551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.064218 (+ 12667us) server_negotiation.cc:407] Connection header received
0927 18:44:18.064902 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.064911 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.065029 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.065141 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:18.066663 (+ 1522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35340: BlockingRecv error: recv got EOF from 127.0.0.1:35340 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260927 18:44:18.067649 25107 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:18.051436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.2.95.126:45031)
0927 18:44:18.051710 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.051717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.063772 (+ 12055us) server_negotiation.cc:407] Connection header received
0927 18:44:18.066026 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.066034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.066132 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.066200 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:18.067383 (+ 1183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50134: BlockingRecv error: recv got EOF from 127.0.0.1:50134 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 18:44:18.069901 25106 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:18.064775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59770 (local address 127.2.95.124:42629)
0927 18:44:18.065174 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.065181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.068130 (+ 2949us) server_negotiation.cc:407] Connection header received
0927 18:44:18.068368 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.068375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.068491 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.068567 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:18.069601 (+ 1034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59770: BlockingRecv error: recv got EOF from 127.0.0.1:59770 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
W20260927 18:44:18.087265 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:18.084441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.2.95.125:37525)
0927 18:44:18.084684 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.084691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.085287 (+ 596us) server_negotiation.cc:407] Connection header received
0927 18:44:18.085552 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.085559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.085766 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.085890 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:18.086924 (+ 1034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35346: BlockingRecv error: recv got EOF from 127.0.0.1:35346 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 18:44:18.103269 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:18.100579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.2.95.125:37525)
0927 18:44:18.100823 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.100831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.101459 (+ 628us) server_negotiation.cc:407] Connection header received
0927 18:44:18.101751 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.101758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.101877 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.102011 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:18.102979 (+ 968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35358: BlockingRecv error: recv got EOF from 127.0.0.1:35358 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260927 18:44:18.111732 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:18.109068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.2.95.125:37525)
0927 18:44:18.109281 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.109288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.109853 (+ 565us) server_negotiation.cc:407] Connection header received
0927 18:44:18.110165 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.110172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.110306 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.110424 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:18.111438 (+ 1014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35364: BlockingRecv error: recv got EOF from 127.0.0.1:35364 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 18:44:18.135111 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:18.132312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.2.95.125:37525)
0927 18:44:18.132685 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.132693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.133303 (+ 610us) server_negotiation.cc:407] Connection header received
0927 18:44:18.133528 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.133533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.133635 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.133758 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:18.134787 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35370: BlockingRecv error: recv got EOF from 127.0.0.1:35370 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
W20260927 18:44:18.163312 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:18.161113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.2.95.125:37525)
0927 18:44:18.161364 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.161371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.161646 (+ 275us) server_negotiation.cc:407] Connection header received
0927 18:44:18.161893 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.161901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.162016 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.162162 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:18.163011 (+ 849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35376: BlockingRecv error: recv got EOF from 127.0.0.1:35376 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 18:44:18.185587 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:18.183227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.2.95.125:37525)
0927 18:44:18.183499 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.183507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.183919 (+ 412us) server_negotiation.cc:407] Connection header received
0927 18:44:18.184136 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.184142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.184261 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.184399 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:18.185289 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35380: BlockingRecv error: recv got EOF from 127.0.0.1:35380 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260927 18:44:18.224010 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:18.221515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.2.95.125:37525)
0927 18:44:18.221835 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.221843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.222110 (+ 267us) server_negotiation.cc:407] Connection header received
0927 18:44:18.222327 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.222333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.222475 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.222616 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:18.223698 (+ 1082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35382: BlockingRecv error: recv got EOF from 127.0.0.1:35382 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260927 18:44:18.294063 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:18.291992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.2.95.125:37525)
0927 18:44:18.292226 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.292234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.292498 (+ 264us) server_negotiation.cc:407] Connection header received
0927 18:44:18.292747 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.292753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.292848 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.292987 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:18.293810 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35388: BlockingRecv error: recv got EOF from 127.0.0.1:35388 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260927 18:44:18.428808 25109 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:18.426302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.2.95.125:37525)
0927 18:44:18.426558 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.426566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.426954 (+ 388us) server_negotiation.cc:407] Connection header received
0927 18:44:18.427178 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.427183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.427275 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.427380 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:18.428409 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35396: BlockingRecv error: recv got EOF from 127.0.0.1:35396 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 18:44:18.448812 25122 catalog_manager.cc:1259] Loaded cluster ID: bd20bbd8ae4e4ec89506826cde903702
I20260927 18:44:18.448948 25122 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49: loading cluster ID for follower catalog manager: success
I20260927 18:44:18.452153 25122 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49: acquiring CA information for follower catalog manager: success
I20260927 18:44:18.454450 25122 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8dfdc3305e4a461e804e74753ac20a49: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:44:18.524834 25136 catalog_manager.cc:1259] Loaded cluster ID: bd20bbd8ae4e4ec89506826cde903702
I20260927 18:44:18.524991 25136 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6: loading cluster ID for follower catalog manager: success
I20260927 18:44:18.528132 25136 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6: acquiring CA information for follower catalog manager: success
I20260927 18:44:18.530781 25136 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1112378a652c4b6e8376cdc5d06f41d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:44:18.698590 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:18.692149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.2.95.125:37525)
0927 18:44:18.692427 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.692435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.692600 (+ 165us) server_negotiation.cc:407] Connection header received
0927 18:44:18.692858 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.692863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.693067 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.693192 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:18.693664 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:18.694505 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:18.695967 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:18.696418 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:18.696536 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:18.697112 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:18.697884 (+ 772us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:18.698045 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:18.698147 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 18:44:18.698259 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 18:44:18.712373 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:18.705512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.2.95.124:42629)
0927 18:44:18.706078 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.706083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.706130 (+ 47us) server_negotiation.cc:407] Connection header received
0927 18:44:18.706255 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.706259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.706462 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.706625 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:18.707201 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:18.708056 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:18.709644 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:18.710194 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:18.710310 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:18.710917 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:18.711698 (+ 781us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:18.711819 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:18.711920 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 18:44:18.712010 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":321,"threads_started":1}
I20260927 18:44:18.728724 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:18.722216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.2.95.125:37525)
0927 18:44:18.722426 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.722434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.722461 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:44:18.722554 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.722559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.722663 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.722765 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:18.723396 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:18.724293 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:18.726204 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:18.726929 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:18.727037 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:18.727680 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:18.727840 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:18.727906 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:18.728271 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 18:44:18.728371 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 18:44:18.729185 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:18.720959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59784 (local address 127.2.95.124:42629)
0927 18:44:18.721132 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.721137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.722766 (+ 1629us) server_negotiation.cc:407] Connection header received
0927 18:44:18.722897 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.722902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.722985 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.723077 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:18.723703 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:18.724888 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:18.726916 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:18.727543 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:18.727633 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:18.728304 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:18.728436 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:18.728499 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:18.728810 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 18:44:18.728902 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 18:44:18.740538 25242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:18.722021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.2.95.126:45031)
0927 18:44:18.734188 (+ 12167us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:18.734194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:18.734214 (+ 20us) server_negotiation.cc:407] Connection header received
0927 18:44:18.734336 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:18.734341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:18.734582 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:18.734766 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:18.735411 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:18.736240 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:18.738293 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:18.738725 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:18.738829 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:18.739432 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:18.739897 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:18.739995 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:18.740136 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 18:44:18.740220 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11996,"thread_start_us":11829,"threads_started":1}
I20260927 18:44:18.746662 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24887
I20260927 18:44:18.747813 24949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:18.872041 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 24887
I20260927 18:44:18.893728 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 24961
I20260927 18:44:18.895061 25023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:19.019869 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 24961
I20260927 18:44:19.041193 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 25036
I20260927 18:44:19.042634 25098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:19.165867 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 25036
2026-09-27T18:44:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4941 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 18:44:19 dist-test-slave-9jll krb5kdc[25248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 18:44:19 dist-test-slave-9jll krb5kdc[25248](info): set up 2 sockets
Sep 27 18:44:19 dist-test-slave-9jll krb5kdc[25248](info): commencing operation
krb5kdc: starting...
W20260927 18:44:20.238412 2429 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/test-user.keytab.
Sep 27 18:44:20 dist-test-slave-9jll krb5kdc[25248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T18:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T18:44:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.95.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
Entry for principal kudu/127.2.95.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab.
I20260927 18:44:20.565318 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:43583
--webserver_interface=127.2.95.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:43583,127.2.95.125:34545,127.2.95.124:38161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 18:44:20.964895 25261 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:44:20.965356 25261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:20.965461 25261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:20.965598 25261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:20.973707 25261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:20.973855 25261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:20.973953 25261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:20.974023 25261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:20.974115 25261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:20.985355 25261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:43583,127.2.95.125:34545,127.2.95.124:38161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.126.keytab
--principal=kudu/127.2.95.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.126:43583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.95.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:20.987301 25261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:20.988986 25261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:21.003460 25267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:21.003479 25269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:21.003417 25266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:21.004756 25261 server_base.cc:1034] running on GCE node
I20260927 18:44:21.005841 25261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:21.007923 25261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:21.009218 25261 hybrid_clock.cc:648] HybridClock initialized: now 1790534661009132 us; error 53 us; skew 500 ppm
I20260927 18:44:21.009748 25261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:21 dist-test-slave-9jll krb5kdc[25248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:21.013653 25261 init.cc:377] Logged in from keytab as kudu/127.2.95.126@KRBTEST.COM (short username kudu)
I20260927 18:44:21.015552 25261 webserver.cc:460] Webserver started at http://127.2.95.126:34971/ using document root <none> and password file <none>
I20260927 18:44:21.016469 25261 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:21.016577 25261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:21.016950 25261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:21.020753 25261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/data/instance:
uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24"
format_stamp: "Formatted at 2026-09-27 18:44:21 on dist-test-slave-9jll"
server_key: "7d3e482208a76d654d3e312dd9382eef"
server_key_iv: "f98bc2693651df927eec4e75cafa0ba8"
server_key_version: "encryptionkey@0"
I20260927 18:44:21.021762 25261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/wal/instance:
uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24"
format_stamp: "Formatted at 2026-09-27 18:44:21 on dist-test-slave-9jll"
server_key: "7d3e482208a76d654d3e312dd9382eef"
server_key_iv: "f98bc2693651df927eec4e75cafa0ba8"
server_key_version: "encryptionkey@0"
I20260927 18:44:21.028470 25261 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 18:44:21.033228 25276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:21.035475 25261 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 18:44:21.035701 25261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/wal
uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24"
format_stamp: "Formatted at 2026-09-27 18:44:21 on dist-test-slave-9jll"
server_key: "7d3e482208a76d654d3e312dd9382eef"
server_key_iv: "f98bc2693651df927eec4e75cafa0ba8"
server_key_version: "encryptionkey@0"
I20260927 18:44:21.035941 25261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:21.056454 25261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:21.061627 25261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:21.061957 25261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:21.083097 25261 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.126:43583
I20260927 18:44:21.083119 25327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.126:43583 every 8 connection(s)
I20260927 18:44:21.084754 25261 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/data/info.pb
I20260927 18:44:21.089366 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 25261
I20260927 18:44:21.089546 25328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:21.089583 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-0/wal/instance
I20260927 18:44:21.090286 2429 external_mini_cluster.cc:1428] Setting key 57146208228d474f67141b07f31204c5
I20260927 18:44:21.099692 25328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:21 dist-test-slave-9jll krb5kdc[25248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.95.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.125@KRBTEST.COM" created.
I20260927 18:44:21.126029 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.105963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.2.95.126:43583)
0927 18:44:21.106711 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:21.106720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:21.106778 (+ 58us) server_negotiation.cc:407] Connection header received
0927 18:44:21.108179 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:21.108186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:21.108472 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:21.108691 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:21.110211 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.111566 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.113076 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.113733 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.113927 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:21.117861 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:21.117871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:21.117889 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:21.117926 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:21.120971 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:21.121739 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:21.121749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:21.121752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:21.121921 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:21.122324 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:21.122328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:21.122330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:21.123014 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:21.123234 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:21.123854 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 18:44:21.124276 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":329,"threads_started":1}
I20260927 18:44:21.126047 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.104279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:43583 (local address 127.0.0.1:38728)
0927 18:44:21.105287 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.105333 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:21.107726 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:21.108717 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:21.108724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:21.109532 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:21.109978 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:21.110000 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.111736 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.111740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:21.112928 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:21.112947 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.113975 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.113982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:21.114197 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:21.114975 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:21.115006 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:21.117590 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:21.121124 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:21.121137 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:21.121154 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:21.121572 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:21.122029 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:21.122034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:21.122038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:21.122213 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:21.123153 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:21.123162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:21.123393 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 18:44:21.123690 (+ 297us) client_negotiation.cc:240] Negotiation successful
0927 18:44:21.124467 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":166,"thread_start_us":431,"threads_started":1}
I20260927 18:44:21.130594 25328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:44:21.131968 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.131426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59126)
0927 18:44:21.131642 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.131788 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 18:44:21.132344 25280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
I20260927 18:44:21.134622 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.134120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59128)
0927 18:44:21.134317 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.134410 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 18:44:21.134900 25328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:34545: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.125.keytab.
Entry for principal kudu/127.2.95.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.125.keytab.
I20260927 18:44:21.163147 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.125:34545
--webserver_interface=127.2.95.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.125.keytab
--principal=kudu/127.2.95.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:43583,127.2.95.125:34545,127.2.95.124:38161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:44:21.184486 25328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } attempt: 1
I20260927 18:44:21.186173 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.185559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59144)
0927 18:44:21.185842 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.186003 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260927 18:44:21.187316 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.186922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59146)
0927 18:44:21.187087 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.187165 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 18:44:21.187561 25328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:34545: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
I20260927 18:44:21.255856 25328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } attempt: 2
I20260927 18:44:21.257493 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.256899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59158)
0927 18:44:21.257198 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.257307 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 18:44:21.258705 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.258242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59160)
0927 18:44:21.258413 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.258558 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 18:44:21.258901 25328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:34545: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
I20260927 18:44:21.350207 25328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } attempt: 3
I20260927 18:44:21.351653 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.351117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59172)
0927 18:44:21.351370 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.351477 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 18:44:21.353068 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.352565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59174)
0927 18:44:21.352777 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.352880 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 18:44:21.353286 25328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:34545: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
I20260927 18:44:21.496611 25328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } attempt: 4
I20260927 18:44:21.498095 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.497432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59184)
0927 18:44:21.497817 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.497920 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260927 18:44:21.499260 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.498845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59200)
0927 18:44:21.499022 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.499117 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 18:44:21.499468 25328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.125:34545: Network error: Client connection negotiation failed: client connection to 127.2.95.125:34545: connect: Connection refused (error 111)
W20260927 18:44:21.557757 25335 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:44:21.558214 25335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:21.558319 25335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:21.558436 25335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:21.566592 25335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:21.566730 25335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:21.566823 25335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:21.566895 25335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:21.566985 25335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:21.578217 25335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:43583,127.2.95.125:34545,127.2.95.124:38161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.125.keytab
--principal=kudu/127.2.95.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.125:34545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.95.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:21.580041 25335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:21.581704 25335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:21.594705 25340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:21.594624 25341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:21.594662 25343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:21.596204 25335 server_base.cc:1034] running on GCE node
I20260927 18:44:21.597075 25335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:21.598827 25335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:21.600070 25335 hybrid_clock.cc:648] HybridClock initialized: now 1790534661599986 us; error 54 us; skew 500 ppm
I20260927 18:44:21.600462 25335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:21 dist-test-slave-9jll krb5kdc[25248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:21.604238 25335 init.cc:377] Logged in from keytab as kudu/127.2.95.125@KRBTEST.COM (short username kudu)
I20260927 18:44:21.606081 25335 webserver.cc:460] Webserver started at http://127.2.95.125:34231/ using document root <none> and password file <none>
I20260927 18:44:21.606890 25335 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:21.606981 25335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:21.607304 25335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:21.610903 25335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/data/instance:
uuid: "00c558f871eb4eb9b764e207eca26201"
format_stamp: "Formatted at 2026-09-27 18:44:21 on dist-test-slave-9jll"
server_key: "92dc91057d08e22c3c07f07ccca391f9"
server_key_iv: "0c81332a16001a11926c523448084e43"
server_key_version: "encryptionkey@0"
I20260927 18:44:21.611817 25335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/wal/instance:
uuid: "00c558f871eb4eb9b764e207eca26201"
format_stamp: "Formatted at 2026-09-27 18:44:21 on dist-test-slave-9jll"
server_key: "92dc91057d08e22c3c07f07ccca391f9"
server_key_iv: "0c81332a16001a11926c523448084e43"
server_key_version: "encryptionkey@0"
I20260927 18:44:21.619083 25335 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260927 18:44:21.623649 25350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:21.625401 25335 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260927 18:44:21.625617 25335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/wal
uuid: "00c558f871eb4eb9b764e207eca26201"
format_stamp: "Formatted at 2026-09-27 18:44:21 on dist-test-slave-9jll"
server_key: "92dc91057d08e22c3c07f07ccca391f9"
server_key_iv: "0c81332a16001a11926c523448084e43"
server_key_version: "encryptionkey@0"
I20260927 18:44:21.625948 25335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:21.657205 25335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:21.662274 25335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:21.662595 25335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:21.682477 25335 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.125:34545
I20260927 18:44:21.682477 25401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.125:34545 every 8 connection(s)
I20260927 18:44:21.684157 25335 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/data/info.pb
I20260927 18:44:21.686141 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 25335
I20260927 18:44:21.686321 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-1/wal/instance
I20260927 18:44:21.686825 2429 external_mini_cluster.cc:1428] Setting key b8f6bb2f5722c806162dda56e689bbd3
I20260927 18:44:21.688859 25402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:21.697093 25402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:21 dist-test-slave-9jll krb5kdc[25248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.125@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.95.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.95.124@KRBTEST.COM" created.
I20260927 18:44:21.721407 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.700420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38740 (local address 127.2.95.126:43583)
0927 18:44:21.701031 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:21.701038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:21.702338 (+ 1300us) server_negotiation.cc:407] Connection header received
0927 18:44:21.704591 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:21.704598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:21.704696 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:21.704819 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:21.707070 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.707954 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.709642 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.710147 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.710293 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:21.714907 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:21.714913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:21.714917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:21.714966 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:21.717816 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:21.718693 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:21.718698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:21.718701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:21.718835 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:21.719304 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:21.719308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:21.719310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:21.719672 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:21.719834 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:21.720709 (+ 875us) server_negotiation.cc:299] Negotiation successful
0927 18:44:21.720983 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":332,"threads_started":1}
I20260927 18:44:21.722740 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.701139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:43583 (local address 127.0.0.1:38740)
0927 18:44:21.702104 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.702156 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:21.704228 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:21.705238 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:21.705249 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:21.706036 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:21.706806 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:21.706829 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.708132 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.708136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:21.709471 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:21.709485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.710422 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.710430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:21.710718 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:21.711544 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:21.711619 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:21.714586 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:21.717984 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:21.717995 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:21.718019 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:21.718543 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:21.718945 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:21.718951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:21.718955 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:21.719184 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:21.719811 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:21.719820 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:21.720300 (+ 480us) client_negotiation.cc:769] Sending connection context
0927 18:44:21.720583 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 18:44:21.721269 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":395,"threads_started":1}
I20260927 18:44:21.725134 25402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:21 dist-test-slave-9jll krb5kdc[25248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.125@KRBTEST.COM for kudu/127.2.95.125@KRBTEST.COM
I20260927 18:44:21.743902 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.726240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59210)
0927 18:44:21.726490 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.726510 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:21.726649 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:21.727785 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:21.727789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:21.728381 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:21.728552 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:21.728563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.730219 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.730223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:21.731254 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:21.731268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.732821 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.732838 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:21.733087 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:21.733866 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:21.733888 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:21.736623 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:21.741029 (+ 4406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:21.741035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:21.741040 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:21.741516 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:21.741963 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:21.741968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:21.741972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:21.742124 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:21.742846 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:21.742851 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:21.743000 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 18:44:21.743126 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 18:44:21.743481 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 18:44:21.744017 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.726355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.2.95.125:34545)
0927 18:44:21.727151 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:21.727159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:21.727194 (+ 35us) server_negotiation.cc:407] Connection header received
0927 18:44:21.727266 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:21.727276 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:21.727588 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:21.727766 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:21.728728 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.730063 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.731391 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.732447 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.732693 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:21.736927 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:21.736946 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:21.736968 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:21.737008 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:21.740761 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:21.741656 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:21.741717 (+ 61us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:21.741719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:21.741854 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:21.742231 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:21.742236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:21.742239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:21.742688 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:21.742911 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:21.743306 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 18:44:21.743647 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":617,"thread_start_us":453,"threads_started":1}
I20260927 18:44:21.747349 25402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:44:21.748775 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.748142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53278)
0927 18:44:21.748360 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.748578 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 18:44:21.749080 25352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
I20260927 18:44:21.751104 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.750654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53290)
0927 18:44:21.750847 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.750953 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 18:44:21.751362 25402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:38161: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.95.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.124.keytab.
Entry for principal kudu/127.2.95.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.124.keytab.
I20260927 18:44:21.759048 2429 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknccDtC/build/asan/bin/kudu
/tmp/dist-test-tasknccDtC/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.124:38161
--webserver_interface=127.2.95.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.124.keytab
--principal=kudu/127.2.95.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.95.84:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.95.126:43583,127.2.95.125:34545,127.2.95.124:38161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 18:44:21.798751 25328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } attempt: 5
I20260927 18:44:21.800834 25402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } attempt: 1
I20260927 18:44:21.802209 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.801595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53294)
0927 18:44:21.801845 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.801995 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 18:44:21.803378 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.802926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53306)
0927 18:44:21.803085 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.803201 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 18:44:21.804734 25402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:38161: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Sep 27 18:44:21 dist-test-slave-9jll krb5kdc[25248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for kudu/127.2.95.125@KRBTEST.COM
I20260927 18:44:21.817943 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.799823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59214)
0927 18:44:21.800139 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.800165 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:21.800325 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:21.800871 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:21.800877 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:21.801489 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:21.801782 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:21.801801 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.803335 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.803342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:21.804399 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:21.804421 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.805757 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.805770 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:21.806110 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:21.806859 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:21.806881 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:21.809627 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:21.814602 (+ 4975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:21.814608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:21.814611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:21.815188 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:21.815744 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:21.815749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:21.815752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:21.815920 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:21.816859 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:21.816863 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:21.817068 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 18:44:21.817204 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 18:44:21.817581 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260927 18:44:21.818019 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.800047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59214 (local address 127.2.95.125:34545)
0927 18:44:21.800296 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:21.800301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:21.800328 (+ 27us) server_negotiation.cc:407] Connection header received
0927 18:44:21.800481 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:21.800487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:21.800659 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:21.800779 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:21.801960 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.803130 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.804588 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:21.805389 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:21.805544 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:21.810121 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:21.810132 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:21.810138 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:21.810213 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:21.814308 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:21.815354 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:21.815362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:21.815366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:21.815562 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:21.816066 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:21.816073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:21.816077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:21.816690 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:21.816880 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:21.817353 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 18:44:21.817663 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260927 18:44:21.819725 25328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:44:21.821322 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.820738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53310)
0927 18:44:21.820961 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.821127 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 18:44:21.822585 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.822141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53318)
0927 18:44:21.822317 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.822419 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 18:44:21.822803 25328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:38161: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
I20260927 18:44:21.873023 25402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } attempt: 2
I20260927 18:44:21.874053 25328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } attempt: 1
I20260927 18:44:21.874676 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.873994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53330)
0927 18:44:21.874294 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.874432 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 18:44:21.875221 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.874681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53336)
0927 18:44:21.874887 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.874994 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 18:44:21.876174 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.875789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53338)
0927 18:44:21.875938 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.876016 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 18:44:21.876345 25402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:38161: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
I20260927 18:44:21.876400 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.876001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53344)
0927 18:44:21.876185 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.876274 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 18:44:21.876590 25328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:38161: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
I20260927 18:44:21.944864 25328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } attempt: 2
I20260927 18:44:21.946434 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.945798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53354)
0927 18:44:21.946102 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.946222 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 18:44:21.947764 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.947305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53366)
0927 18:44:21.947525 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.947614 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 18:44:21.947994 25328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:38161: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
I20260927 18:44:21.967621 25402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } attempt: 3
I20260927 18:44:21.969120 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.968508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53378)
0927 18:44:21.968767 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.968888 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 18:44:21.970368 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:21.969883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53384)
0927 18:44:21.970129 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:21.970233 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260927 18:44:21.970589 25402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:38161: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
I20260927 18:44:22.054235 25328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } attempt: 3
I20260927 18:44:22.055899 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.055310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53398)
0927 18:44:22.055563 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:22.055693 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 18:44:22.057078 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.056639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53410)
0927 18:44:22.056874 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:22.056954 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260927 18:44:22.057299 25328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:38161: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
I20260927 18:44:22.113835 25402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } attempt: 4
I20260927 18:44:22.115263 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.114680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53424)
0927 18:44:22.114966 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:22.115069 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 18:44:22.116413 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.116056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53434)
0927 18:44:22.116218 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:22.116294 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 18:44:22.116621 25402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:38161: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
W20260927 18:44:22.158141 25410 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 18:44:22.158583 25410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 18:44:22.158686 25410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 18:44:22.158804 25410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 18:44:22.166917 25410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 18:44:22.167047 25410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 18:44:22.167152 25410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 18:44:22.167224 25410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 18:44:22.167313 25410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 18:44:22.178522 25410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.95.84:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.95.126:43583,127.2.95.125:34545,127.2.95.124:38161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/krb5kdc/kudu_127.2.95.124.keytab
--principal=kudu/127.2.95.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.95.124:38161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.95.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 17:40:55 UTC on ccycloud.quasar-girbhv.root.comops.site
build id 4676276
ASAN enabled
I20260927 18:44:22.180392 25410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 18:44:22.182046 25410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 18:44:22.195371 25415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:22.195340 25418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 18:44:22.195412 25416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 18:44:22.196616 25410 server_base.cc:1034] running on GCE node
I20260927 18:44:22.197518 25410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 18:44:22.199316 25410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 18:44:22.200585 25410 hybrid_clock.cc:648] HybridClock initialized: now 1790534662200496 us; error 63 us; skew 500 ppm
I20260927 18:44:22.200930 25410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 18:44:22 dist-test-slave-9jll krb5kdc[25248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534662, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 18:44:22.204411 25410 init.cc:377] Logged in from keytab as kudu/127.2.95.124@KRBTEST.COM (short username kudu)
I20260927 18:44:22.206228 25410 webserver.cc:460] Webserver started at http://127.2.95.124:35099/ using document root <none> and password file <none>
I20260927 18:44:22.207029 25410 fs_manager.cc:359] Metadata directory not provided
I20260927 18:44:22.207121 25410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 18:44:22.207475 25410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 18:44:22.211090 25410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/data/instance:
uuid: "e4cc7b0ec8a64019bdf23b935ff01b50"
format_stamp: "Formatted at 2026-09-27 18:44:22 on dist-test-slave-9jll"
server_key: "80e4a076e0f71dda84f7e271e9efcacb"
server_key_iv: "7eeb1e2f5e181bfcada0cdaa1b20d059"
server_key_version: "encryptionkey@0"
I20260927 18:44:22.211941 25410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/wal/instance:
uuid: "e4cc7b0ec8a64019bdf23b935ff01b50"
format_stamp: "Formatted at 2026-09-27 18:44:22 on dist-test-slave-9jll"
server_key: "80e4a076e0f71dda84f7e271e9efcacb"
server_key_iv: "7eeb1e2f5e181bfcada0cdaa1b20d059"
server_key_version: "encryptionkey@0"
I20260927 18:44:22.217965 25410 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 18:44:22.221901 25425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 18:44:22.223582 25410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 18:44:22.223783 25410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/data,/tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/wal
uuid: "e4cc7b0ec8a64019bdf23b935ff01b50"
format_stamp: "Formatted at 2026-09-27 18:44:22 on dist-test-slave-9jll"
server_key: "80e4a076e0f71dda84f7e271e9efcacb"
server_key_iv: "7eeb1e2f5e181bfcada0cdaa1b20d059"
server_key_version: "encryptionkey@0"
I20260927 18:44:22.224023 25410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 18:44:22.234520 25328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } attempt: 4
I20260927 18:44:22.235875 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.235310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53448)
0927 18:44:22.235602 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:22.235709 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 18:44:22.237035 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.236659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53458)
0927 18:44:22.236811 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:22.236883 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 18:44:22.237336 25328 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.95.124:38161: Network error: Client connection negotiation failed: client connection to 127.2.95.124:38161: connect: Connection refused (error 111)
I20260927 18:44:22.244735 25410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 18:44:22.249755 25410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 18:44:22.250056 25410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 18:44:22.271566 25410 rpc_server.cc:307] RPC server started. Bound to: 127.2.95.124:38161
I20260927 18:44:22.271546 25476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.95.124:38161 every 8 connection(s)
I20260927 18:44:22.273293 25410 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/data/info.pb
I20260927 18:44:22.278071 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 18:44:22.281934 2429 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknccDtC/build/asan/bin/kudu as pid 25410
I20260927 18:44:22.282130 2429 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknccDtC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790534284715420-2429-0/minicluster-data/master-2/wal/instance
I20260927 18:44:22.282626 2429 external_mini_cluster.cc:1428] Setting key aace8a5ccadd37f0aeddc85bc3c5e0e1
I20260927 18:44:22.287247 25477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:22 dist-test-slave-9jll krb5kdc[25248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.124@KRBTEST.COM
Sep 27 18:44:22 dist-test-slave-9jll krb5kdc[25248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534662, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.124@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:44:22.307485 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.291218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.2.95.126:43583)
0927 18:44:22.291851 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:22.291857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:22.292410 (+ 553us) server_negotiation.cc:407] Connection header received
0927 18:44:22.292645 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:22.292650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:22.292748 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:22.292950 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:22.294191 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.295025 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.296638 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.297312 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.297474 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.301247 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:22.301254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:22.301259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:22.301308 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:22.304213 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.305071 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.305077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.305080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.305247 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.305620 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.305623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.305625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.305939 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:22.306090 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:22.306780 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 18:44:22.307071 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":282,"threads_started":1}
I20260927 18:44:22.308779 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.284829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.2.95.124:38161)
0927 18:44:22.285749 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:22.285769 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:22.285831 (+ 62us) server_negotiation.cc:407] Connection header received
0927 18:44:22.288085 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:22.288121 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:22.288791 (+ 670us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:22.289288 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:22.290834 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.292872 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.294486 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.295310 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.295515 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.300459 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:22.300470 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:22.300489 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:22.300530 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:22.303595 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.304504 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.304514 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.304517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.304715 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.305090 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.305094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.305096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.305907 (+ 811us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:22.306226 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:22.306593 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 18:44:22.307124 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":614,"thread_start_us":371,"threads_started":1}
I20260927 18:44:22.308815 25482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.291200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.126:43583 (local address 127.0.0.1:38750)
0927 18:44:22.292230 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:22.292292 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:22.292483 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:22.293002 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:22.293010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:22.293745 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:22.294030 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:22.294039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.295196 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.295201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:22.296472 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:22.296484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.297541 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.297546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:22.297718 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.298419 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:22.298447 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:22.301029 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:22.304356 (+ 3327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:22.304366 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:22.304379 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:22.304952 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:22.305357 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:22.305361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:22.305365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:22.305532 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:22.306066 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:22.306078 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:22.306306 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 18:44:22.306572 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 18:44:22.307262 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":164,"thread_start_us":543,"threads_started":1}
I20260927 18:44:22.310889 25477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 18:44:22 dist-test-slave-9jll krb5kdc[25248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534662, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.124@KRBTEST.COM for kudu/127.2.95.125@KRBTEST.COM
Sep 27 18:44:22 dist-test-slave-9jll krb5kdc[25248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.126@KRBTEST.COM
I20260927 18:44:22.327584 25482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.311878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.125:34545 (local address 127.0.0.1:59222)
0927 18:44:22.312060 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:22.312082 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:22.312221 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:22.312842 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:22.312847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:22.313275 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:22.313444 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:22.313454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.314707 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.314712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:22.315836 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:22.315848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.316727 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.316733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:22.316940 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.317744 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:22.317766 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:22.320211 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:22.324277 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:22.324283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:22.324288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:22.324905 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:22.325376 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:22.325381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:22.325385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:22.325617 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:22.326369 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:22.326375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:22.326629 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 18:44:22.326791 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 18:44:22.327155 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 18:44:22.327973 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.311879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.2.95.125:34545)
0927 18:44:22.312092 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:22.312097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:22.312148 (+ 51us) server_negotiation.cc:407] Connection header received
0927 18:44:22.312411 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:22.312417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:22.312516 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:22.312596 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:22.313612 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.314568 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.315981 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.316585 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.316690 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.320439 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:22.320448 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:22.320453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:22.320515 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:22.324104 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.325032 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.325039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.325043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.325226 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.325780 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.325785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.325789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.326242 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:22.326406 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:22.327014 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 18:44:22.327448 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:44:22.329581 25477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } has no permanent_uuid. Determining permanent_uuid...
I20260927 18:44:22.330822 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.312324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.2.95.126:43583)
0927 18:44:22.312511 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:22.312516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:22.312532 (+ 16us) server_negotiation.cc:407] Connection header received
0927 18:44:22.312624 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:22.312628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:22.312727 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:22.312823 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:22.314169 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.315004 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.316433 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.317006 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.317144 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.322023 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:22.322058 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:22.322063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:22.322139 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:22.326731 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.328382 (+ 1651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.328388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.328392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.328583 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.329061 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.329065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.329068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.329450 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:22.329798 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:22.330126 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 18:44:22.330416 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 18:44:22 dist-test-slave-9jll krb5kdc[25248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534662, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.124@KRBTEST.COM for kudu/127.2.95.124@KRBTEST.COM
Sep 27 18:44:22 dist-test-slave-9jll krb5kdc[25248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.95.125@KRBTEST.COM
I20260927 18:44:22.346501 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.330974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.2.95.124:38161)
0927 18:44:22.331128 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:22.331133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:22.331150 (+ 17us) server_negotiation.cc:407] Connection header received
0927 18:44:22.331530 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:22.331536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:22.331603 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:22.331694 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:22.332678 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.333652 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.335243 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.335913 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.336055 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.339824 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:22.339831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:22.339836 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:22.339878 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:22.343090 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.343928 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.343934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.343937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.344085 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.344653 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.344660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.344664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.345182 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:22.345357 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:22.345798 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 18:44:22.346066 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":39}
I20260927 18:44:22.346566 25482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.330631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53466)
0927 18:44:22.330824 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:22.330841 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:22.331402 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:22.331715 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:22.331720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:22.332305 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:22.332516 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:22.332529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.333806 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.333812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:22.335096 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:22.335109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.336068 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.336072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:22.336349 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.337032 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:22.337056 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:22.339612 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:22.343249 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:22.343256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:22.343261 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:22.343797 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:22.344223 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:22.344227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:22.344229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:22.344451 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:22.345312 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:22.345316 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:22.345458 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 18:44:22.345699 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 18:44:22.346223 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":179}
I20260927 18:44:22.351148 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.332724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.2.95.125:34545)
0927 18:44:22.332911 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:22.332916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:22.332971 (+ 55us) server_negotiation.cc:407] Connection header received
0927 18:44:22.333029 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:22.333035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:22.333113 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:22.333219 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:22.334908 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.336104 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.337786 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.338231 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.338326 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.344054 (+ 5728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:22.344064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:22.344069 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:22.344118 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:22.347753 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.348784 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.348790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.348793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.348983 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.349498 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.349502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.349513 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.349920 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:22.350197 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:22.350611 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 18:44:22.350836 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 18:44:22.353792 25477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50: Bootstrap starting.
I20260927 18:44:22.357064 2429 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 18:44:22.358688 25477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:22.360239 25477 log.cc:824] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:22.359900 25303 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38752
I20260927 18:44:22.364254 25477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50: No bootstrap required, opened a new log
I20260927 18:44:22.377002 25477 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } }
I20260927 18:44:22.377607 25477 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cc7b0ec8a64019bdf23b935ff01b50, State: Initialized, Role: FOLLOWER
I20260927 18:44:22.378315 25477 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } }
I20260927 18:44:22.379568 25484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } } }
I20260927 18:44:22.379966 25484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:22.381049 25477 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:44:22.393402 25495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:44:22.393602 25495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:44:22.389904 25410 master_runner.cc:186] Error getting master registration for 127.2.95.126:43583: OK, instance_id { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" instance_seqno: 1790534661077227 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:44:22.400864 25410 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 18:44:22.415870 25402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } attempt: 5
Sep 27 18:44:22 dist-test-slave-9jll krb5kdc[25248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.125@KRBTEST.COM for kudu/127.2.95.124@KRBTEST.COM
I20260927 18:44:22.434867 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.417169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.2.95.124:38161)
0927 18:44:22.417388 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:22.417395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:22.417424 (+ 29us) server_negotiation.cc:407] Connection header received
0927 18:44:22.417566 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:22.417571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:22.417799 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:22.417935 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:22.419140 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.420401 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.422498 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.423218 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.423361 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.427663 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:22.427673 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:22.427679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:22.427737 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:22.431335 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.432357 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.432363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.432365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.432531 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.433033 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.433038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.433042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.433536 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:22.433713 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:22.434201 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 18:44:22.434463 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 18:44:22.434861 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.416887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53482)
0927 18:44:22.417211 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:22.417240 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:22.417433 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:22.417987 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:22.417993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:22.418636 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:22.418939 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:22.418954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.420584 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.420590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:22.422317 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:22.422338 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.423389 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.423395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:22.423646 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.424423 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:22.424446 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:22.427343 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:22.431501 (+ 4158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:22.431508 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:22.431513 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:22.432192 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:22.432700 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:22.432705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:22.432710 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:22.432898 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:22.433660 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:22.433702 (+ 42us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:22.433879 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 18:44:22.434018 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 18:44:22.434400 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 18:44:22.442754 25402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201: Bootstrap starting.
I20260927 18:44:22.446985 25402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:22.448511 25402 log.cc:824] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:22.451943 25402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201: No bootstrap required, opened a new log
I20260927 18:44:22.459951 25402 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } }
I20260927 18:44:22.460352 25402 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c558f871eb4eb9b764e207eca26201, State: Initialized, Role: FOLLOWER
I20260927 18:44:22.461024 25402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } }
I20260927 18:44:22.462284 25498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } } }
I20260927 18:44:22.462754 25498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:22.463815 25402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 18:44:22.476698 25509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 18:44:22.477012 25509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 18:44:22.472391 25335 master_runner.cc:186] Error getting master registration for 127.2.95.126:43583: OK, instance_id { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" instance_seqno: 1790534661077227 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 18:44:22.484254 25335 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 18:44:22.502655 25431 tablet.cc:2369] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:22.514598 25328 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } attempt: 5
Sep 27 18:44:22 dist-test-slave-9jll krb5kdc[25248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790534661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.95.126@KRBTEST.COM for kudu/127.2.95.124@KRBTEST.COM
I20260927 18:44:22.531371 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.515583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.2.95.124:38161)
0927 18:44:22.515800 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:22.515807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:22.516140 (+ 333us) server_negotiation.cc:407] Connection header received
0927 18:44:22.516232 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:22.516238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:22.516355 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:22.516445 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:22.517374 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.518593 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.519859 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.520614 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.520734 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.524724 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:22.524729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:22.524733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:22.524773 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:22.528144 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.528924 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.528931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.528935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.529123 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.529601 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.529624 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.529628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.530125 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:22.530298 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:22.530688 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 18:44:22.530923 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:44:22.531363 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.515519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.95.124:38161 (local address 127.0.0.1:53486)
0927 18:44:22.515841 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 18:44:22.515862 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 18:44:22.516017 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 18:44:22.516500 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 18:44:22.516505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 18:44:22.516994 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 18:44:22.517187 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:22.517198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.518758 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.518763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:22.519712 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 18:44:22.519726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.520786 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.520792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 18:44:22.520998 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.521726 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 18:44:22.521748 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 18:44:22.524516 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 18:44:22.528291 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:22.528298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:22.528303 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:22.528816 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:22.529264 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 18:44:22.529269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 18:44:22.529274 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 18:44:22.529491 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 18:44:22.530257 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 18:44:22.530263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 18:44:22.530430 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 18:44:22.530561 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 18:44:22.530899 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260927 18:44:22.538925 25328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24: Bootstrap starting.
I20260927 18:44:22.543889 25328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24: Neither blocks nor log segments found. Creating new log.
I20260927 18:44:22.546710 25328 log.cc:824] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24: Log is configured to *not* fsync() on all Append() calls
I20260927 18:44:22.551240 25328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24: No bootstrap required, opened a new log
I20260927 18:44:22.559664 25328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } }
I20260927 18:44:22.560139 25328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3b5e115fab4dd6bb6ceafd5075ac24, State: Initialized, Role: FOLLOWER
I20260927 18:44:22.560760 25328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } }
I20260927 18:44:22.562214 25512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } } }
I20260927 18:44:22.563002 25512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:22.563778 25328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 18:44:22.563861 25303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 18:44:22.564098 25303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [term 0 FOLLOWER]: Advancing to term 1
W20260927 18:44:22.566099 25282 tablet.cc:2369] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:22.570063 25303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } }
I20260927 18:44:22.574162 25377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "00c558f871eb4eb9b764e207eca26201"
I20260927 18:44:22.574859 25377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:22.579181 25377 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3b5e115fab4dd6bb6ceafd5075ac24 in term 1.
I20260927 18:44:22.580374 25280 leader_election.cc:302] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c558f871eb4eb9b764e207eca26201, 1d3b5e115fab4dd6bb6ceafd5075ac24; no voters:
I20260927 18:44:22.582618 25452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50"
I20260927 18:44:22.583292 25452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [term 0 FOLLOWER]: Advancing to term 1
I20260927 18:44:22.587867 25452 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3b5e115fab4dd6bb6ceafd5075ac24 in term 1.
I20260927 18:44:22.588867 25303 leader_election.cc:288] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [CANDIDATE]: Term 1 election: Requested vote from peers 00c558f871eb4eb9b764e207eca26201 (127.2.95.125:34545), e4cc7b0ec8a64019bdf23b935ff01b50 (127.2.95.124:38161)
I20260927 18:44:22.596376 25512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 18:44:22.610193 25512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [term 1 LEADER]: Becoming Leader. State: Replica: 1d3b5e115fab4dd6bb6ceafd5075ac24, State: Running, Role: LEADER
I20260927 18:44:22.611320 25512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } }
I20260927 18:44:22.638489 25519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3b5e115fab4dd6bb6ceafd5075ac24. Latest consensus state: current_term: 1 leader_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } } }
I20260927 18:44:22.639329 25519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:22.641623 25532 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 18:44:22.642596 25533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 18:44:22.657579 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.617590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.2.95.124:38161)
0927 18:44:22.617932 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:22.617939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:22.626319 (+ 8380us) server_negotiation.cc:407] Connection header received
0927 18:44:22.626676 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:22.626684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:22.626819 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:22.626902 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:22.629395 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.630599 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.643257 (+ 12658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.644054 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.644246 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.648561 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:22.648571 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:22.648576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:22.648639 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:22.652997 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.653993 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.654004 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.654008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.654192 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.654652 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.654663 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.654666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.655103 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:22.655394 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:22.656798 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0927 18:44:22.657121 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
W20260927 18:44:22.665968 25356 tablet.cc:2369] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 18:44:22.672391 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.618287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.2.95.126:43583)
0927 18:44:22.618490 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:22.618497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:22.629149 (+ 10652us) server_negotiation.cc:407] Connection header received
0927 18:44:22.629921 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:22.629929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:22.630046 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:22.630134 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:22.645302 (+ 15168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.646633 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.654676 (+ 8043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.655394 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.655503 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.662570 (+ 7067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:22.662579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:22.662584 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:22.662643 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:22.668271 (+ 5628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.669351 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.669357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.669361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.669566 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.670136 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.670141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.670144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.670624 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:22.670973 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:22.671635 (+ 662us) server_negotiation.cc:299] Negotiation successful
0927 18:44:22.671923 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 18:44:22.674767 25533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 18:44:22.676280 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:22.621599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.2.95.125:34545)
0927 18:44:22.621849 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:22.621855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:22.631255 (+ 9400us) server_negotiation.cc:407] Connection header received
0927 18:44:22.632102 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:22.632109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:22.632221 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:22.632308 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:22.649072 (+ 16764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.650213 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.656918 (+ 6705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:22.657701 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:22.657822 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:22.662580 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 18:44:22.662587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 18:44:22.662591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 18:44:22.662643 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 18:44:22.666775 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.673327 (+ 6552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.673335 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.673340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.673556 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 18:44:22.674086 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 18:44:22.674092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 18:44:22.674095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 18:44:22.674534 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 18:44:22.674836 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:22.675550 (+ 714us) server_negotiation.cc:299] Negotiation successful
0927 18:44:22.675883 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 18:44:22.686693 25377 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [term 1 FOLLOWER]: Refusing update from remote peer 1d3b5e115fab4dd6bb6ceafd5075ac24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:22.686693 25452 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [term 1 FOLLOWER]: Refusing update from remote peer 1d3b5e115fab4dd6bb6ceafd5075ac24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 18:44:22.687750 25512 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:22.688931 25519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 18:44:22.722203 25498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3b5e115fab4dd6bb6ceafd5075ac24. Latest consensus state: current_term: 1 leader_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } } }
I20260927 18:44:22.722602 25498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:22.725541 25484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3b5e115fab4dd6bb6ceafd5075ac24. Latest consensus state: current_term: 1 leader_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } } }
I20260927 18:44:22.725903 25484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:22.730294 25535 mvcc.cc:205] Tried to move back new op lower bound from 7334029978349101056 to 7334029978114457600. Current Snapshot: MvccSnapshot[applied={T|T < 7334029978349101056}]
I20260927 18:44:22.740854 25498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } } }
I20260927 18:44:22.741214 25498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:22.750417 25484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } } }
I20260927 18:44:22.750792 25484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 18:44:22.760210 25519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } } }
I20260927 18:44:22.760625 25519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:22.761224 25533 catalog_manager.cc:1347] Generated new cluster ID: 290f384c85444649a54d09a1db16fb46
I20260927 18:44:22.761354 25533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 18:44:22.761073 25519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3b5e115fab4dd6bb6ceafd5075ac24" member_type: VOTER last_known_addr { host: "127.2.95.126" port: 43583 } } peers { permanent_uuid: "00c558f871eb4eb9b764e207eca26201" member_type: VOTER last_known_addr { host: "127.2.95.125" port: 34545 } } peers { permanent_uuid: "e4cc7b0ec8a64019bdf23b935ff01b50" member_type: VOTER last_known_addr { host: "127.2.95.124" port: 38161 } } }
I20260927 18:44:22.762562 25519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24 [sys.catalog]: This master's current role is: LEADER
I20260927 18:44:22.851596 25533 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 18:44:22.853822 25533 catalog_manager.cc:1504] Loading token signing keys...
I20260927 18:44:22.875018 25533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d3b5e115fab4dd6bb6ceafd5075ac24: Generated new TSK 0
I20260927 18:44:22.879350 25533 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 18:44:23.041977 25481 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:23.025263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.2.95.126:43583)
0927 18:44:23.025581 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.025589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.031194 (+ 5605us) server_negotiation.cc:407] Connection header received
0927 18:44:23.039723 (+ 8529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.039733 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.039887 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.039990 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:23.041655 (+ 1665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38760: BlockingRecv error: recv got EOF from 127.0.0.1:38760 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260927 18:44:23.043303 25480 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:23.026708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.2.95.124:38161)
0927 18:44:23.026961 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.026968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.041194 (+ 14226us) server_negotiation.cc:407] Connection header received
0927 18:44:23.041579 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.041586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.041728 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.041821 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:23.043030 (+ 1209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53512: BlockingRecv error: recv got EOF from 127.0.0.1:53512 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260927 18:44:23.046671 25407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:23.032833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.2.95.125:34545)
0927 18:44:23.033101 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.033107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.041872 (+ 8765us) server_negotiation.cc:407] Connection header received
0927 18:44:23.042138 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.042145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.042261 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.042337 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:23.046378 (+ 4041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59250: BlockingRecv error: recv got EOF from 127.0.0.1:59250 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 18:44:23.069798 25407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:23.066638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.2.95.125:34545)
0927 18:44:23.066892 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.066899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.067352 (+ 453us) server_negotiation.cc:407] Connection header received
0927 18:44:23.067642 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.067649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.067813 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.067929 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:23.069403 (+ 1474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59266: BlockingRecv error: recv got EOF from 127.0.0.1:59266 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 18:44:23.080195 25407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:23.075802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.2.95.125:34545)
0927 18:44:23.076040 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.076047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.078375 (+ 2328us) server_negotiation.cc:407] Connection header received
0927 18:44:23.078616 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.078623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.078737 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.078857 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:23.079911 (+ 1054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59268: BlockingRecv error: recv got EOF from 127.0.0.1:59268 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 18:44:23.117007 25407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:23.114297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.2.95.125:34545)
0927 18:44:23.114552 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.114560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.115241 (+ 681us) server_negotiation.cc:407] Connection header received
0927 18:44:23.115504 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.115510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.115647 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.115771 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:23.116715 (+ 944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59284: BlockingRecv error: recv got EOF from 127.0.0.1:59284 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 18:44:23.127859 25407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:23.125534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.2.95.125:34545)
0927 18:44:23.125851 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.125857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.126159 (+ 302us) server_negotiation.cc:407] Connection header received
0927 18:44:23.126401 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.126406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.126490 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.126594 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:23.127541 (+ 947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59292: BlockingRecv error: recv got EOF from 127.0.0.1:59292 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
W20260927 18:44:23.145246 25407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:23.141104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59294 (local address 127.2.95.125:34545)
0927 18:44:23.141414 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.141421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.142871 (+ 1450us) server_negotiation.cc:407] Connection header received
0927 18:44:23.143286 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.143293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.143416 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.143599 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:23.144914 (+ 1315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59294: BlockingRecv error: recv got EOF from 127.0.0.1:59294 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 18:44:23.169023 25407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:23.166274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.2.95.125:34545)
0927 18:44:23.166572 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.166579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.167056 (+ 477us) server_negotiation.cc:407] Connection header received
0927 18:44:23.167331 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.167336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.167443 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.167636 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:23.168723 (+ 1087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59306: BlockingRecv error: recv got EOF from 127.0.0.1:59306 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260927 18:44:23.208158 25407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:23.205758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.2.95.125:34545)
0927 18:44:23.206017 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.206025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.206472 (+ 447us) server_negotiation.cc:407] Connection header received
0927 18:44:23.206675 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.206682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.206800 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.206945 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:23.207825 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59320: BlockingRecv error: recv got EOF from 127.0.0.1:59320 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 18:44:23.278558 25407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:23.276403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.2.95.125:34545)
0927 18:44:23.276605 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.276610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.276977 (+ 367us) server_negotiation.cc:407] Connection header received
0927 18:44:23.277164 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.277171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.277288 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.277418 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:23.278277 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59322: BlockingRecv error: recv got EOF from 127.0.0.1:59322 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 18:44:23.396199 25495 catalog_manager.cc:1259] Loaded cluster ID: 290f384c85444649a54d09a1db16fb46
I20260927 18:44:23.396337 25495 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50: loading cluster ID for follower catalog manager: success
I20260927 18:44:23.399448 25495 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50: acquiring CA information for follower catalog manager: success
I20260927 18:44:23.401726 25495 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4cc7b0ec8a64019bdf23b935ff01b50: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 18:44:23.412822 25407 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 18:44:23.410666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.2.95.125:34545)
0927 18:44:23.410909 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.410917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.411296 (+ 379us) server_negotiation.cc:407] Connection header received
0927 18:44:23.411541 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.411546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.411636 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.411746 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 18:44:23.412594 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59328: BlockingRecv error: recv got EOF from 127.0.0.1:59328 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 18:44:23.479821 25509 catalog_manager.cc:1259] Loaded cluster ID: 290f384c85444649a54d09a1db16fb46
I20260927 18:44:23.479990 25509 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201: loading cluster ID for follower catalog manager: success
I20260927 18:44:23.483331 25509 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201: acquiring CA information for follower catalog manager: success
I20260927 18:44:23.485584 25509 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00c558f871eb4eb9b764e207eca26201: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 18:44:23.679136 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:23.673370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.2.95.125:34545)
0927 18:44:23.673603 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.673608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.673783 (+ 175us) server_negotiation.cc:407] Connection header received
0927 18:44:23.673990 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.673995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.674195 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.674340 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:23.674753 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:23.675525 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:23.676993 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:23.677396 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:23.677502 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:23.678007 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:23.678605 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:23.678732 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:23.678775 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 18:44:23.678869 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 18:44:23.695768 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:23.687847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.2.95.124:38161)
0927 18:44:23.690128 (+ 2281us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.690134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.690153 (+ 19us) server_negotiation.cc:407] Connection header received
0927 18:44:23.690292 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.690296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.690484 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.690662 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:23.691113 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:23.691873 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:23.693298 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:23.693769 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:23.693908 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:23.694458 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:23.695047 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:23.695191 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:23.695305 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 18:44:23.695407 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2132,"thread_start_us":1975,"threads_started":1}
I20260927 18:44:23.711905 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:23.702791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.2.95.125:34545)
0927 18:44:23.702972 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.702977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.704759 (+ 1782us) server_negotiation.cc:407] Connection header received
0927 18:44:23.705120 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.705127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.705232 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.705300 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:23.706227 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:23.707409 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:23.709380 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:23.710005 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:23.710130 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:23.711013 (+ 883us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:23.711166 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:23.711271 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:23.711446 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 18:44:23.711563 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 18:44:23.713933 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:23.703264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.2.95.124:38161)
0927 18:44:23.703498 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.703505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.707548 (+ 4043us) server_negotiation.cc:407] Connection header received
0927 18:44:23.707660 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.707666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.707835 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.707918 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:23.708483 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:23.709512 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:23.711817 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:23.712464 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:23.712591 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:23.713036 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:23.713147 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:23.713229 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:23.713380 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 18:44:23.713471 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 18:44:23.715273 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 18:44:23.703145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.2.95.126:43583)
0927 18:44:23.703786 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0927 18:44:23.703794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 18:44:23.707693 (+ 3899us) server_negotiation.cc:407] Connection header received
0927 18:44:23.707898 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 18:44:23.707904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 18:44:23.708159 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 18:44:23.708362 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 18:44:23.709109 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:23.710111 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:23.712260 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 18:44:23.712869 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 18:44:23.712997 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 18:44:23.713802 (+ 805us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 18:44:23.714371 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 18:44:23.714461 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 18:44:23.714771 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 18:44:23.714947 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":271,"threads_started":1}
I20260927 18:44:23.721467 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 25261
I20260927 18:44:23.722728 25323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:23.835875 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 25261
W20260927 18:44:23.858632 25427 connection.cc:535] server connection from 127.0.0.1:53486 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 18:44:23.859649 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 25335
I20260927 18:44:23.860896 25397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:23.985579 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 25335
I20260927 18:44:24.005177 2429 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknccDtC/build/asan/bin/kudu pid 25410
I20260927 18:44:24.006639 25472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 18:44:24.118664 2429 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknccDtC/build/asan/bin/kudu with pid 25410
2026-09-27T18:44:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4952 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9894 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (379360 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST